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Introduction 


As part of the Discovery Program’s continuous improvement effort, a 
Discovery Program Lessons-Leamed workshop was designed to review how 
well the Discovery Program is moving toward its goal of providing low-cost 
research opportunities to the planetary science community while ensuring 
continued U.S. leadership in solar system exploration. The principal focus of 
the workshop was on the recently completed Announcement of Opportunity 
(AO) cycle, but program direction and program management were also open 
to comment. The objective of the workshop was to identify both the strengths 
and weaknesses of the process up to this point, with the goal of improving the 
process for the next AO cycle. 

The process for initializing the workshop was to solicit comments from 
the communities involved in the program and to use the feedback as the basis 
for establishing the workshop agenda. By approaching the workshop in this 
manner, we were able to focus the workshop on the areas of highest interest. 
The workshop was designed to provide an opportunity for participants from 
each of the constituent groups (academia, industry, and government organiza- 
tions) to present their own lessons learned, followed by four working sessions 
to consider issues relevant to each area and derive potential improvements to 
the process. The following four sessions were developed after reviewing and 
synthesizing both the formal feedback received and informal feedback ob- 
tained during discussions with various participants: ( 1 ) Science and Return 
on Investment; (2) Technology vs. Risk; Mission Success and Other Factors; 
(3) Cost; (4) AO/ AO Process Changes and Program Management. 

The chair of the workshop was Dr. David Bohlin from the Office of Space 
Science’s Space Physics Division. Dr. Bohlin was the Chair of the Space 
Science Steering Committee, which reviewed the initial Discovery AO pro- 
cess to assure it was conducted properly. His previous experiences brought a 
well-informed independent approach to the workshop. We were also fortu- 
nate to have leading experts from each of the constituencies present their 
lessons learned as part of the initial workshop activities. These presentations 
were representative of the feedback received, positive and constructive, and 
highlighted the areas of greatest interest. The presenters focused their views 
on the areas that both enhanced their efforts to develop their proposals and 
those that inhibited their work, with the intention of ensuring that positive 
attributes were retained and the areas of concern were reviewed for improve- 
ments. Headquarters also presented lessons learned. Copies of each workshop 
presentation are included in this volume. 

Following the presentations, workshop attendees separated into the four 
splinter group sessions to address the issues identified for that group, priori- 
tize them, and develop potential resolutions to the most important ones. Each 
group was chaired by a representative from the communities represented who 
served as the spokesperson for the group. The sessions were extremely 
productive and drove straight to the heart of the issues. Crosscutting presen- 
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tations were given in the morning on the second day, followed by further 
splinter group meetings. At the end of the morning the session chairs pre- 
sented their preliminary findings. 

The splinter group’s presentations identified the principle issues discussed, 
potential solutions to the issues, and a number of alternative recommenda- 
tions. These presentations revealed the interrelated issues among the different 
groups that must be integrated and synthesized into a coherent approach for 
them to be viable. To facilitate bringing the splinter group recommendations 
to closure, the workshop participants concurred with the establishment of a 
small steering group, composed of the four splinter group chairs plus several 
other representatives, to continue the work initiated at the workshop. The 
steering group was formed and assigned the task of completing their activities 
by the end of summer. Dr. William Boynton from the University of Arizona 
agreed to chair the steering group, and the membership included representa- 
tives from each of the three principal constituencies: academia, industry, and 
government organizations. The steering group was given the responsibility of 
adjudicating most of the comments and recommendations identified during 
the workshop and forming a complete integrated set of improvements for 
NASA to consider. The conclusions of the steering group are included in this 
report. 

The activities of the Lessons-Learned workshop are having far-reaching 
effects on the Discovery Program as well as other similar programs in NASA. 
The issues, discussions, and preliminary recommendations of the workshop 
are being used in the Explorer programs and in new programs such as the 
Mission to Planet Earth’s new program. Earth System Science Pathfinders. 

As all these programs use the same foundation of experiences, individual 
program implementation strategies are expected to converge, ultimately 
making the task of proposing to each different AO a similar process. 

The success of this workshop is a direct result of input from the commu- 
nity, and the Office of Space Science wishes to thank all contributors to this 
process. 

Logistics and administrative and publications support for this workshop 
were provided by the staff of the Lunar and Planetary Institute. 
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Program 


Wednesday, June 14, 1995 


8:00 a.m. Introduction, Agenda 


Hampton Room 


8:15 a.m. Program Status and Future Outlook 

8:35 a.m. Science Community Perspectives 

1 . W. Boynton (Univ. of Arizona) 

2. C. Russell (UCLA) 

3. A. Binder (Lockheed/Martin) 


9:45 a.m. Industry Perspectives 

1 . D. Roalstad (Ball Aerospace) 

2. B. Clark (Lockheed/Martin) 

3. G. Adams (Hughes) 

4. J. Freitag (TRW) 

5. D. Smith (Boeing) 

6. D. Tenerelli (Lockheed/Martin) 

7. D. Gump (LunaCorp) 

1:00 p.m. Government / FFRDC Perspectives 

1. S. Hubbard (ARC) 

2. E. Davis (JPL) 

3. L. Crawford (Applied Physics Lab) 

2:00 p.m. Summary of Written Community Comments 
NASA Headquarters Perspectives 

2:45 p.m. Splinter Group Objectives and Assignments 

3:30 p.m. Splinter Group Sessions 

1 . Science and Return on 
Investments 

2. Technology vs. Risk: Mission 
Success and Other Factors 

3. Cost 

4. AO / AO Process Changes and 
Program Management 


Calvert Room 

Embassy Room 
Senate Room 

Council Room 


— David Bohlin 
— Jurgen Rahe 


— Mark Saunders 
— Mark Saunders 

— Mark Saunders 

— William Boynton (Chair) 

— David Roalstad (Chair) 

— Lawrence Mitchler (Chair) 

— Scott Hubbard (Chair) 


♦CHARTER: Develop recommendations for next AO* 
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Thursday, June 15, 1995 

8:00 a.m. Splinter Group Sessions 

1 . Science and Return on Investments 

Calvert Room 

— William Boynton (Chair) 

2. Technology vs. Risk: Mission 
Success and Other Factors 

Embassy Room 

— David Roalstad (Chair) 

3. Cost 

Senate Room 

— Scott Hubbard (Chair) 

4. AO/AOO Process Changes and 
Program Management 

Council Room 

— Lawrence Mitchler (Chair) 

10:00 a.m. Plenary Session Splinter Group Reports 
and Discussions 

Empire Room 


12:00 p.m. Conclusions 


— David Bohlin, 

12:30 p.m. Adjourn 


Mark Saunders 
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Steering Group Recommendations 


These recommendations of the Discovery Lessons-Leamed 
Steering Group follow from a meeting in Washington DC on 
July 1 9 and a conference call on September 26. The member- 
ship of the steering group is William Boynton (University of 
Arizona, Lunar and Planetary Laboratory), Chair; Kevin 
Baines (Jet Propulsion Laboratory); Mary Chiu (Johns Hopkins 
University/ Applied Physics Laboratory); Cynthia Faulconer 
(Lockheed Martin); Scott Hubbard (NASA Ames Research 
Center); Larry Mitchler (TRW Civil & International Systems 
Division); David Roalstad (Ball Corporation, Electro-Opti- 
cal Subsystems); and Christopher Russell (UCLA, Institute 
of Geophysics & Planetary Physics). 

ASSUMPTIONS 

This section specifically does not contain the recommen- 
dations of the group, though we may agree with them. It states 
the ground rules of the environment in which we were work- 
ing. 

1. Though proposals could be grouped based on similar 
cost and evaluated independent of cost within that group, at 
some point low-cost proposals would have to compete with 
high-cost proposals with cost being a significant discrimina- 
tor. 

2. The new AO process will be the final selection; it will 
not select multiple proposals to be downselected later. 

3. In the event the full $ 1 50M is not available, the AO will 
state that it was targeted to proposals below a certain cost or 
spending profile, but the AO will never target just high-cost 
proposals (see rule number 1). 

RECOMMENDATIONS 

1. The proposal evaluation process shall take two steps, 
(a) The selection of successful proposals in the first step shall 
be based mainly on cost and a detailed science evaluation 
(including instrumentation). Information in the areas of tech- 
nical approach and management shall also be evaluated to 
determine feasibility of the project, (b) The first step is a 
mandatory gate; only proposals selected by step 1 may be 
submitted for step 2. (c) The formula to be used for evaluation 
in step 1 shall be similar to the one used in step 2 as noted 
below. 

2. Though a mathematical formula is proposed for the 
selection score, it is understood that the AA for Space Science 
will be able to use discretion to adjust for programmatic or 
other factors. 

3. The formula for rating proposals following step 2 shall 
be: 


Score * ROI x POS 

where 

ROI * return on investment; 

POS *= probability of success; 

(0.8 x Science x Science Risk + 0.1 xPA + O.lxEO) 
(Total Cost to NASA) 

PA « public awareness; 

EO = educational opportunity 

The science, PA, and EO scores shall be normalized by 
dividing all scores by the highest one in that category, i.e., the 
scores will range from 0 to 1 in each of these categories. (See 
recommendation 8 for details on applying science risk fac- 
tors.) 

Technology and small and small-disadvantaged business 
contracting plans shall not be part of the evaluation criteria. 

The Total Cost to NASA is the proposed cost for all phases, 
A through E. It shall include direct costs, civil-servant costs, 
and launch-vehicle costs. Independent cost estimates deter- 
mined by the review process shall be used only for accessing 
cost risk. 

The probability of success (POS) is determined by combin- 
ing risk factors in the area of cost, technical approach, and 
management. The POS should describe the probability that 
the mission will succeed in meeting the cost and schedule 
targets and that it will succeed in delivering and servicing the 
science payload as required. (The probability that the science 
will be achieved is contained in the Science Risk term and is 
not considered here.) The individual risk factors are numbers 
ranging from 0 to 1 , and they are to express the probability that 
the factor will allow the ROI to be to be realized, e.g., a cost 
risk factor of 0.98 indicates that there is little chance of a cost 
overrun; a technical risk of 0.3 indicates that the technical 
approach has serious flaws that make it unlikely that the 
mission will succeed. It should be understood that the risks in 
these areas are not independent, i.e., a poor management 
approach would certainly contribute to a cost risk. The evalu- 
ation process should combine all of these risks into the POS 
term, the probability of achieving the ROI. 

4. For the first step in the process, the formula shall be 
modified such that PA, EO, and cost risk are not part of the 
evaluation. In addition, the combined weight of management 
risk and technical risk shall be reduced so that it cannot modify 
the ROI by more than a factor of 2 and technical risk shall be 
about twice as important as management risk. These criteria 
can be satisfied by limiting the dynamic range of technical risk 
to 0.4 (ranging from 0.6 to 1 .0) and of management risk to 0.2 
(from 0.8 to 1 .0). Thus the formula for rating proposals follow- 
ing step 1 shall be: 
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_ (Science x Science Risk) x (modified POS) 

Score = 

(Total Cost to NASA) 


where 

modified POS - (0.6 + Technical Risk x 0.4) x 
(0.8 + Management Risk x 0.2) 


5. The cost estimate in the step-one proposal should be 
considered to be a good-faith estimate of what the step-two 
proposal costs will be. Any growth between the first and 
second proposal shall have to be justified, and if not well 
justified, will contribute to an assessment of additional cost 
risk in evaluation of the step-two proposal. In no case may the 
costs increase by more than 15% between the first and second 
proposal. No form SF 141 1 shall be required in the first-step 
evaluation. 

6. The level of cost detail for step 2 shall be comparable 
to that requested in the first Discovery AO (see Table 1 ). The 
form SF 141 1 shall be required only for the first phase to be 
funded, normally phase A. 

7. The cost detail for step 1 shall consist of only the three 
exhibits required in the step-2 proposal. 

8. The science evaluation shall not be based on simple 
adjectival grades. It should be scored using the science objec- 
tives listed in NASA and COMPLEX strategies as a guide. 
The score should reflect the number of objectives addressed, 
the importance of the objectives, and the thoroughness with 
which the objectives are addressed or answered. Determining 
the relative importance of different objectives may not always 
be easy, but it is noted that COMPLEX has consistently 
recommended a strategy of reconnaissance, followed by ex- 
ploration, followed by intensive study with progressively 
greater science return from the more detailed investigations. 
The idea is that the later missions can build on the results of 
the earlier missions and return much higher science, but only 
if the groundwork is laid such that the more detailed mission 
can be designed to answer well-posed questions. Within these 
three categories, the COMPLEX and NASA strategies often 
ascribe priorities to different objects in the solar system, and 
to different objectives for a given object. These strategies can 
only be considered guides, and substantial discretion will 
need to be left to the scientific review panel. For example, 
because these documents may become dated based on U.S. or 
foreign missions that started after the documents were pub- 
lished, the panel can significantly reduce the importance of 
highly rated science that can be reasonably expect to be 
provided by another currently approved mission. 

This score can be open ended in the sense that the more 
objectives satisfied, the higher the score. If one mission 
addresses twice as many objectives as another and they are 
addressed as well and are of equal importance, then that 
mission should score twice as high as the other mission. 

The science risk reflects the probability that the science 
generated by the mission will ultimately be the science pro- 
posed. This risk should be assessed independent of the risks 


in the POS (discussed in recommendation #3). It should 
include risks associated with the instrumentation, the ability 
of the measurements to address the proposed science, the 
quality of the data management and archiving plan, and the 
quality of the science team. In order to permit the addition of 
one or more high-risk objectives to a mission, the science risk 
should be assessed and applied separately for each objective. 
For example, if a mission is proposed to an object that ad- 
dresses two objectives very well and a third will only be 
addressed if some high-risk approach works, its science rank- 
ing would be higher than a similar mission that only proposed 
the same two objectives that could be achieved with little risk. 
(It should be noted, however, that if the reviewers determin- 
ing the POS found that too much of management’s attention 
was focused on the third objective, the overall rating could be 
lower.) 

9. A suggested guideline for the AO schedule is: 

❖ Step-1 proposals due 3 months after AO 

❖ Step-1 results announced 2 months after due date 

❖ Step-2 proposals due 2 months after results announced 

❖ Final selection 3 months after step-2 due date 

TABLE 1. Suggested step-two Discovery 
cost proposal contents. 


Exhibits 

1. Total program cost elements by phase by FY. (1) 

2. Major tasks, total payload, and total spacecraft cost 
elements by phase by FY. 

3. Flight subsystems (one level below total spacecraft) 
and instruments (one level below total payload) cost 
elements by phase by FY. 

Supporting Data 

1. Basis of estimates for exhibits 2 and 3. (2) 

2. Major subcontractor data summary. (3) 

3. Direct rates. (4) 

4. Indirect rates. (5) 

5. Other supporting data. (6) 

Notes: 

(1) Cost elements are labor hours, labor dollars, related payroll 
expenses, overhead, other direct costs, materials and material 
burden, major subcontracts, other subcontracts, general and ad- 
ministrative, and ICOM. 

(2) The SOW defines the work/products to be provided for the 
CBS cell. Describe the method used to estimate the labor hours 
and cost, materials, and subcontracts. For hardware, include an 
equipment list for items greater than $1000 in value defining the 
major components and sources and necessary cost detail. 

(3) Subcontractor summary includes task, identity, type, location, 
amount, contract type, adjustments and burdens. 

(4) Direct rates by FY. 

(5) Indirect rates by FY. 

(6) Data necessary to evaluate proposal not provided elsewhere; 
such as conversions, escalation methods, cost centers, etc. 
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Minutes of the DLLSG Meeting 


Attendees: William Boynton , David Roalstad , Larry 

Mitchler , Scott Hubbard, Cindy Faulconer, Mary Chiu , Mark 
Saunders 

General background information was provided by Mark 
Saunders on the current NASA environment and on recent 
feedback on the Discovery Program provided by D. Goldin, 
A. Diaz, and W. Huntress. While these comments are not 
included in this summary, the overwhelming theme was that 
the Discovery Program was to emphasize low cost — a $ 1 50M 
program proposal would have to be EXCEPTIONAL. In 
addition, there would be no science “targeting” of proposals. 
Competition would be open to all disciplines, with both 
inherently less expensive science competing directly with 
inherently more expensive science. 

M. Saunders stated that the following will be in the next 
proposal request: Statement of Work for each Phase and 
Incentive Plan. The Discovery Program may not have $ 1 SOM 
available for each Announcement of Opportunity (AO). 
Saunders has stated that he would put in the amount available 
for each AO in the future, i.e., if only $75M is available, this 
will be stated up front in the AO along with a phasing of the 
funding. 

W. Boynton suggested that the group initially plan to 
discuss three major topics: ( 1 ) Costs, what information (i.e., 
level of detail) is needed for proposal. (2) ROI equation 
(Return on Investment). (3) AO process (key point — one- or 
two-step process). 

It quickly became clear that these three major topics are all 
interrelated and the discussions would have to iterate through 
these topics rather than make stand-alone decisions within 
these areas. The following summarizes the steering group 
meeting discussions in these three categories as well as 
conclusions about the Opportunity activities. 

The group agreed that September 21 is the goal for com- 
pleting the DLLSG activities. 

AO PROCESS 

A key decision was made fairly early was that the AO 
process would complete the competition phase. (In the first 
AO process there was a competitive phase A — this would not 
be the case in the next AO.) This decision was made to some 
degree based on W. Huntress’ desire not to have another 
phase A competitive downselection, and was supported by 
the DLLSG. The rationale was that everyone wanted a defi- 
nite decision made that would allow the winner to go forward 
and the non winners to move on to the next opportunity. 
Eliminate lingering lower-level effort, which tends to be 
inefficient and nonproductive. 


The question of targeting the AO to a specific area was 
considered. The consensus was that if NASA determines in 
advance that there are reasons to do so, they should so state 
the areas of emphasis in the AO, but that the emphasis should 
not be exclusionary. The ability of the selection officer to 
exercise discretion when a clear case could be made was 
thought to be able to deal with most situations. 

The group agreed that NASA needs to be as blunt as 
possible about available budgets and the desired type of 
mission, and should provide quantitative evaluation criteria 
so that industry does not waste valuable resources preparing 
proposals that will not get selected. 

One-Step or Two-Step Process: The group concurred 

on the following overall goals of any change to the AO 
process: (1) Encourage growth, more participation (more 
proposals). (2) Lower life cycle costs.(3) Get “best” proposals 
possible. (4) Timeliness should also be considered. 

Without defining the details of a two-step process, argu- 
ments could be (and were) made that #2 and #3 could be 
achieved by either a one- or two-step process. However, all 
agreed that given the decision that the AO process would 
decide the winner, only a two-step process would allow # 1 
above. Since the AO process ends the competition, enough 
information has to be given in the proposal to go directly to 
a contract. By definition then, the cost information, plans, 
statement of work, and so on must be detailed. Detail infor- 
mation drives up the cost of proposing, which can reduce the 
number of proposers (i.e., limited to those who can afford to 
spend large amounts of internal funds for B&P). 

There were arguments that life-cycle costs could be re- 
duced by either a one-step or a two-step process. It is true that 
if the two-step is not defined in such a way that the first step 
of the two-step process is at a significantly reduced cost, then 
lower life-cycle costs may not be achieved. (While not stated 
explicitly during the discussions, it is assumed here that the 
information content of the two-step process must be the same 
as that of the one-step process. The challenge is to define the 
minimum content needed for the first step of the two-step 
process.) 

As an aside, with reflecting upon this later, there are some 
simple ways to test lower life-cycle costs. They rely on many 
assumptions, which can result in many scenarios. Three 
sample scenarios are given in the attached appendix. 

While there are an infinite number of scenarios that can be 
imagined, the ones in the appendix illustrate a key point: A 
two-step process, even one with a relatively expensive first 
step (half of expected one-step process cost), reduces total 
life -cycle costs to the proposers as long as the total number of 
proposers stays constant. The most likely way a one-step 
process reduces the life-cycle costs is by reducing the number 
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of proposers — an exclusionary measure that is directly counter 
to promoting growth and participation of the overall commu- 
nity. 

It is still a valid point that a grossly ill-defined two-step 
process could result in increased life-cycle costs (to proposers 
as a group). The information content of the first step has to be 
reduced to decrease the cost of preparation — the minimum 
goal should be to reduce the first step of a two-step process to 
one-half of a one-step proposal. 

A decision was made to use a two-step process (contingent 
on not defining a overly costly first step of the two-step 
process). 

The group felt that the number of missions to proceed to 
step 2 should be in the range of six or seven missions. The 
group, however, also recommended that reason and logic be 
used. In other words, if five are closely spaced in the evalu- 
ation with all others way behind, then go with five. If eight are 
closely spaced, etc., you may want to go with eight. The idea 
was to keep the number of step-2 proposals large enough to 
ensure that interesting and implementable missions make it 
through but not too large that the AO LCC is big. 

Contents of First and Second Steps: The definition of 
the contents of the first and second steps of the two-step 
process did not progress to final conclusions, but general 
agreement was reached on the basic contents. Three basic 
options were considered in the beginning for the first step: (1 ) 
Science only evaluation. (2) Science and technical approach 
evaluation (no cost or management). (3) Mini proposal. 

While not fully discussed for definite decision, the general 
consensus seemed to be that the first step would be to 
downselect, not just rank proposals. It also seemed to be 
general consensus that a mini-proposal with emphasis on the 
science was the best, and a full science peer review panel was 
required to evaluate step-one proposals. However, there still 
needs to be a mechanism to ensure that the mission would not 
be more costly than presented. This meant that step one needs 
to include some level of detail on mission implementation 
and design in order to determine if the science mission was 
feasible and do-able. This generally requires the PI/industry/ 
center teams to be formed and presented in step one. Based on 
these conclusions, there was a general consensus that the 
same evaluation criteria should be used for both steps one and 
two. 

Bogeys were selected for page count of the step one pro- 
posal for each of the major topics as follows: 

❖ Science including instrumentation, 25 pages 

❖ 'Technical Approach including mission design, 10 pages 

❖ Cost information, 5 pages 

❖ Management Plan, 5 pages 

The recommendation for the step-two proposal focused on 
a proposal content similar to the original AO, particularly 
since the outcome would be a mission selected for flight. This 


proposal, though, would build on the step-one proposal, 
particularly in the Science criteria. The content of the pro- 
posal might be as follows: 

❖ Full up proposal with only changes to science (maybe 
100 pages) 

❖ More detail on implementation and design, manage- 
ment, and cost 

❖ Smaller science panel to evaluate 

❖ Evaluation based on ROI formula (same as step-one 
formula) 

❖ Orals and questions as part of step-two evaluation (if 
necessary) 

❖ Proposals must show all deltas (changes) from step one 
and step two. This is advised to avoid “buy-in” in step 
one. 

Timeliness: Some discussion took place on the time 

needed for the two-step process. Following bogeys were dis- 
cussed but not finalized: 

❖ First-Step Responses 3 months after AO release 

❖ First-Step Evaluation Completion 2 months after 
responses due 

❖ Second-Step Responses 2 months after downselect 

❖ Second-Step Selection 3 months after responses due 

❖ Total time from AO release to final selection 1 0 months 

COST DETAIL 

There was some discussion on the cost detail needed for 
both the first and second step of the two-step process. There 
was general consensus that the cost detail for the first step 
should be limited to a budgetary estimate from the proposer 
with backup information at the discretion of the proposer up 
to five pages. Level of detail, e.g., WBS level, was not de- 
cided. L. Mitchler was to provide some detailed suggestions 
on the final cost detail needed through second step. Mitchler’ s 
response is attached (see section on Annotated Cost Instruc- 
tions). 

There was some discussion on penalties if the cost estimate 
provided for the second step was much greater than that of the 
first step. However, it was recognized that penalties were 
difficult to define. A higher cost has a built-in penalty given 
the ROI definition, which will be discussed below. However, 
there seemed to be some concern that a proposer could “buy- 
in” with a lower cost to get through the first step, and prevent 
a more realistically costed proposal getting to the second step. 
Blatant “low-ball” costs can most likely be flagged by review- 
ers and this should not be a real problem. However, “optimis- 
tic” costs vs. “pessimistic” costs may be difficult to distin- 
guish given the fact that the first step is designed to contain 
less detail using a proposer’s budgetary estimate. By nature, 
these estimates are probably only good to about 25%, and this 
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must be recognized by the reviewers — that is, cost-risk evalu- 
ation for the first step must be very coarse. Saunders stated 
that it is probably best to downselect from the first step based 
on the top competitive range, rather than have an absolute 
maximum or minimum number to go to the second step. 
Given a competitive range selection and a coarse cost-risk 
criteria, an “optimistically” costed proposal could be within 
the competitive range of a “pessimistically” costed proposal, 
given that all other evaluation factors are the same. This 
would depend in part on the coarse cost-risk criteria. In fact, 
this might be a good litmus test for the coarse cost-risk criteria 
development. Saunders stated that there could also be a 
statement within the AO that allowed reviewers to down- 
grade proposals that had significant increases in cost between 
the first-step proposal and the second-step proposal. This 
allows the reviewers some latitude for evaluating cost changes 
between the two steps, while putting proposers on notice that 
such changes must be explained and could be counted against 
them. Realistically, with the ROI, any proposer with “opti- 
mistic” costs in the first step would be weeded out with higher 
costs in the second step given consistent science scores. Such 
a proposer only incurs more internal cost without winning 
final selection — not a sound business practice. This is a tough 
problem, and there may not be a perfect solution. 

While not discussed at any length, a proposer’s past per- 
formance should be given due consideration for the cost risk 
assessment. A copy of the latest was provided to Saunders and 
Mitchler. 

RETURN ON INVESTMENT (ROI) 

Boynton handed out a write-up of the definition of the 
ROI. There was general consensus that the ROI was a good 
way to ensure that only proposals that were of high quality 
across ALL categories would receive high scores and that 
“ROI” should be the basis for evaluation in each step of the 
AO process. (There was some perception stated at the work- 
shop that a lower science score could be masked by higher 
scores in other categories such as cost. This should not be the 
case with a multiplicative scoring system.) 

Boynton’s proposal for Return on Investment was repre- 
sented by the following formula: 

(0.8 x Science Value x SciRisk + 0.1 x Ed + 0.1 x PA) 
Total Cost to NASA 

The definition of each of the above criteria is defined 
below. 

Science Value: Science value is to be determined pri- 

marily by how well the proposed science objectives address 
the priorities determined by the strategy of the National 
Research Council (NRC) Committee on Planetary and Lunar 
Exploration (COMPLEX) and NASA’s strategic plans. The 
score should reflect the breadth of the proposed investigation 
(how many objectives are addressed), the depth of the inves- 
tigation (how thoroughly the objectives are addressed), the 


importance of the investigation (the priority of the objectives 
that are addressed), and the quality to which the objectives are 
addressed (the extent to which the proposed measurements 
are sufficient to address the objectives). 

Another more subjective factor to include is the antici- 
pated impact that the investigation will have on the field. It 
is suggested that the value for this factor be put on a linear 
scale with a high-quality mission (such as Voyager or Apollo) 
being 10 and a modest mission (such as Clementine) being 1 . 
This is not a scale from 1 to 10; missions could have values 
greater than 10 or less than 1. 

In order to use these scores in the algorithm, it will need 
to be normalized to unity. This can be accomplished by 
dividing all scores by the score of the highest ranked mission. 

The way cost should be entered into the equation was still 
up for debate. One alternative was to have several (approxi- 
mately three) bins of comparable cost, and once a mission was 
in a particular bin, cost would no longer be a discriminator 
against other missions in the same cost bin. 

Science Risk (SciRisk): This term should be expressed 
as something that approximates a probability (from zero to 
one) and is determined primarily from instrumentation and 
the quality of the science team. 

Technology Infusion was discussed as it relates to the 
science objectives and was agreed that technology infusion 
should not be given a separate score. The rationale for this is 
that the use of the new technology should already manifest 
itself by increasing the science or reducing the risk and would 
thus get into the score with an appropriate weight. 

Ed and PA: These are Educational Program Activities 

and Public Awareness and their evaluation scores would be 
similar to the original AO. 

FINAL EVALUATION SCORE 

The final evaluation score should be a combination of ROI 
and Mission Risks based on the following formula: 

Score - ROI x Risk 

where: 

Risk - Management Risk x Cost Risk x 
Technical Approach Risk 

Like science risk, these three terms should be expressed on 
a scale from zero to one and should be in the form of a 
probability. These terms will describe the probability that the 
criteria will cause the anticipated Return on Investment not 
to be achieved. 

This scoring system does factor in costs at the 50% level. 
While this may not be what some parts of the community 
wants to hear, it is consistent with the current realities of 
NASA. And if the final algorithm to be used is provided in the 
AO, then everyone at least knows how the determination is 
being made. It also uses an assessment of science “value” for 
selection purposes, a theme that has always been associated 
with the Discovery Program. 
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OPPORTUNITY 

The group agreed that the evaluation process must find a 
way to balance the technology vs. risk issue so that proposals 
are not up-checked for new technology/technology insertion 
and down-checked for risk at the same time. As mentioned in 
the ROI discussion, the group felt that technology should not 
be a separate evaluation factor since technology infusion 
would be driven by the science needs. Missions accomplish- 
ing great science by using new technology would be rewarded 
in their science grade. Risks associated with the new technol- 
ogy would be considered in the technical approach evalua- 
tion. 

The group felt that technology transfer was a factor that 
did not enhance the overall evaluation and should be elimi- 
nated. 

In general there was a consensus that that the SB/SDB 
factor imposes solutions that may not be optimal or most cost 
effective and the factor should also be eliminated. 

The group also seemed to agree that the education and 
public awareness factors should be evaluated independently 
and included in the return on investment calculation. 

OUTSTANDING QUESTIONS, ISSUES, 

AND CONCERNS 

As noted in the text above, there were no final conclusions 
drawn by the DLLSG. The group agreed that further work 
would be done through telecons and e-mail until a final 
overall consensus could be reached. The members agreed that 
the next discussion should take place in mid August. 

Several questions, issues, and concerns were expressed 
that would need further consideration. These were: 

❖ Does the two-step process really reduce the life cycle 
cost of the AO? 

❖ Will the two-step process take too long to implement? 

❖ Will the two-step process let people through step one 
that shouldn’t get through? And will it inadvertently 
shut out missions that really should progress to step 
two? 

❖ Should the evaluation formula be multiplicative or 
additive? We discussed the idea that with “additive” 
some missions can survive that have major holes (be- 
cause they make up points in other areas). 

APPENDIX 

COMPARISON OF TOTAL COSTS FOR 

ONE-STEP AND TWO-STEP AO PROCESSES 
FOR VARIOUS SCENARIOS 


may have different ways of viewing these. . . any alternatives 
would be interesting to discuss. I just used these to give me 
some insight into the sensitivities of various factors. 

Scenario 1: Assume number of proposers are the same 
regardless of AO process — choose 28; assume that a 
downselect occurs after the first step; choose 10 for second 
step. Choose an average proposal cost of $1M. Assume that 
the first-step cost of the two-step process is half the cost of a 
one-step process. 

One-step process: Proposals cost $1M to each proposer, 
total cost is 28 x $1M =■ $28M. 

Two-step process : First step cost $0.5M to each proposer, 
first-step cost is 28 x $0.5M = $14M. Second-step cost is 
$0.5M to each proposer, 10 x $0.5M - $5M. Total cost for 
two-step - $19M. 

Total costs to proposers as a group are lower in this 
scenario for two-step. 

Scenario 2: Assume number of proposers are differ- 
ent — that number of proposers willing to risk $1M in the 
beginning is subset of 28— choose 1 4. Other assumptions the 
same as Scenario 1 . 

One-step process: Proposals cost $1M each, total cost is 
14 x $1M - $14M. 

Two-step process: First-step cost is S0.5M each, cost is 28 
x $0.5M - $14M. Second-step cost is $0.5M each, cost is 10 
x $0.5M - $5M. Total cost for two-step is S19M. 

Total costs to proposers as a group are lower for the one- 
step process, but this is achieved only by the reduced number 
of proposers willing to “risk” the higher costs — a smaller 
“group.” This is an exclusionary scenario that directly coun- 
teracts one of the goals of the AO process, which is to promote 
growth and participation. 

Scenario 3: Same as Scenario 1 but with an increase in 
cost for the second step of proposal. Assume number of 
proposers are the same regardless of AO process — choose 28; 
assume that a downselect occurs after first step; choose 1 0 for 
second step. 

One-step process: Proposals cost $1M to each proposer, 
total cost is 28 x $1M - $28M. 

Two-step process: First-step cost is $0.5M to each 
proposer, first-step cost is 28 x $0.5M - $14M. Second-step 
cost is $0.75M to each proposer, 10 x $0.75M * $7.5M. 
Total - $21. 5M. 

Again, total costs to proposers as a group is less for the two- 
step process. However, it illustrates a very possible scenario 
in which the downselect winners incur more individual costs 
in a two-step process vs. a one- step. However, the increased 
costs per individual proposers are not necessarily mandatory 
and are somewhat under the proposer’s control. 


Below are some simplified calculations for evaluating 
relative costs of the one- and two-step AO process. Others 
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Annotated Cost Instructions 


This copy of the Discovery AO cost instructions is 
annotated with questions and suggestions. In our opin- 
ion , the instructions are generally straightforward and 
appropriate for the second step of the next Discovery 
Program competition. We ha ve made suggestions where 
we thought it would reduce the proposer's efforts or ease 
the evaluation process while still providing adequate 
data for evaluation. 

II. COST PLAN 

The cost plan should provide information on the antici- 
pated costs for all phases of the mission. It should also 
describe the plans for tracking and controlling costs, or 
reference the applicable portions of Volume II or the Manage- 
ment Approach section. 

The inflation index provided in Appendix E should be 
used to calculate all real-year dollar amounts, unless an 
industry forward pricing rate is used. If something other than 
the inflation index is used, the rates used should be docu- 
mented. 

A* Preliminary Analysis (Phase A) Cost Estimate. This 
section provides a detailed cost proposal for performing the 
Phase A Study. Detailed plans for the study should be de- 
scribed, but reference may be made to the Technical Ap- 
proach Section of Volume II. 

In completing this section, the following instructions will 
apply: 

1. Contract Pricing Proposal . 

(a) The cost proposal It will include, as a summary of 
total proposed Phase A costs, a completed SF 141 1, as in- 
cluded in Appendix C. 

It is our understanding of the FARs that a 14 1 1 is only 
required if there is no competition. In the case of Discov- 
ery there most likely will be much competition during 
both steps. A 1411 triggers certain cost proposal re- 
quirements regardless of the text of the cost proposal 
instructions. The 1411 may require more cost data than 
necessary or desirable. 

(b) The SF 1411 must be signed by the proposer's autho- 
rized representative. 

No comment. 

2. Cost Elements Breakdown. To effectively evaluate 
the Phase A cost proposal, NASA requires costs and support- 
ing evidence stating the basis for the estimated costs. The 
proposal will include, but is not limited to: 


(a) Direct Labor. 

( 1 ) Explain the basis of labor-hour estimates for each 
of the labor classifications. Are labor classifications 
really useful? 

(2) State the number of productive work-hours per 
month. 

No comment. 

(3) Provide a schedule of the direct labor rates used in 
the proposal. Discuss the basis for developing the pro- 
posed direct labor rates including the cost centers in- 
volved; the forward-pricing method (including mid- 
point, escalation factors, anticipated impact of future 
union contracts, etc.); and element included in the 
rates, such as overtime, shift differential, incentives, 
allowances, etc. 

Cost centers needs to be defined. The term is am- 
biguous. We suggest that cost center be defined as a 
company , a NASA center, a university, etc . That is, each 
member of a team be defined as a cost center. 

(4) If available, submit evidence of Government ap- 
proval of direct labor rates for proposal purposes for 
each labor classification for the proposed performance 
period. 

No comment. 

(5) If Civil Servant labor is to be used in support of the 
Phase A study, but is not to be charged directly to the 
investigation, then this labor must be considered as a 
contribution by a domestic partner, subject to the same 
restrictions as other contributions by domestic or for- 
eign partners (i.e., the sum of such contributions should 
not exceed approximately one-third of the Phase C/D 
development cost to NASA). 

Trying to price contributions from civil servants was a 
problem. If total cost to NASA is to be the criteria then 
this contribution must be estimated. Perhaps the next 
A O could provide a formula based on Civil Servant labor 
hours. 

(b) Direct Material Submit a breakdown of material 
and parts, including basis for estimates and sources of supply, 
if known. Describe any pricing factors added to material 
prices, such as scrap, rework usage, etc. 
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Is this detail necessary? We suggest just a total 
materials cost for each priced CBS cell. 

(c) Subcontracts. Identify fully each effort (task, item, 
etc.) to be subcontracted, and list the selected subcontractors, 
locations, amount proposed and types of contracts. Explain 
the adjustments, if any, and the indirect rates (or burdens) 
applied to the subcontractors’ proposed amounts. Describe 
fully the cost analysis or price analysis and the negotiations 
conducted regarding the proposed subcontracts. 

This level of detail seems appropriate for major sub- 
contracts (<$500K), but not for all subcontracts. 

(d) Other Direct Costs . 

(1) Travel, Relocation, and Related Costs, (a) Indi- 
cate destination, number of work-trips, duration, and 
purpose. Specify total proposed cost of each trip. 

te this detail useful? We suggest number of work- 
trips, duration , and purpose along with total cost for all 
trips not individual trip cost. 

(b) Explain or submit current company policy regard- 
ing the reimbursement of travel and relocation costs 
and the accounting treatment of such costs as direct 
costs or indirect expenses. Submit copies of Govern- 
ment approvals of such policies, as appropriate. 

No comment. 

(2) Computer. Describe the type of computer, the ex- 
tent of usage, the rates, and the amounts. Explain where 
associated labor costs (programmers, operators, etc.) 
are included in the proposal. 

This requirement seems to be a hangover from the 
old mainframe days . Nowadays , with rare exceptions , 
we only use PCs. Suggest that we just include the cost 
as a line item under ODC. 

(3) Consultants. Indicate the specific task area or prob- 
lem requiring consultant services. Identify the pro- 
posed consultants, and state the quoted daily rate, the 
estimated number of days, and associated costs (such as 
travel), if any. State whether the consultant has been 
compensated at the quoted rate for similar services 
performed in connection with Government contracts. 

No comment. 

(4) Other. Explain and support any other direct costs 
included in the Phase A proposal in a manner similar 
to that described above. 


No comment. 

(e) Indirect Costs. 

(1) List all indirect expense rates and their respective 
cost centers used in the proposal. Indirect expense rates 
(in the context of this AO) include labor overhead, 
material overhead, general and administrative (G&A) 
expenses, and any other cost proposed as an allocation 
to the proposed direct costs. 

No comment. 

(2) If the proposal includes support services for which 
off-site burden rates are used, provide a schedule of the 
off-site burden rates. Include a copy of the company 
policy regarding off- site vs. on-site effort. 

No comment. 

(3) If available, submit evidence of Government ap- 
proval of any/all projected indirect rates for the pro- 
posed period of performance. Indicate the status of rate 
negotiations with the cognizant Government agency, 
and provide a comparative listing of approved bidding 
rates and negotiated actual rates for the past five (5) 
fiscal years. 

No comment. 

(4) Identify separately any independent research and 
development expenses included in the G&A rate. 

This requirement is unclear. We do not want to pro- 
vide JR&D details in the proposal. Our G&A rates , which 
do include JR&D , are negotiated and approved by the 
Government. 

3. Phase A Time-Phased Summary. Prepare a summary 
of the total Phase A estimated costs summarized by cost 
elements and time-phased by month. Note that direct labor 
hours and rates should be shown by category (e.g., engineer- 
ing, manufacturing, etc.). Overhead (or fringe) applied to this 
labor may be shown by cost category or in total. Materials 
amount and subcontract amount should include burdens, as 
appropriate, and should be shown in total. Other direct costs 
should be shown in total. G&A and other indirect costs (such 
as internal research and development charges) should be 
shown as appropriate. 

Is the cost spread by labor category by month really 
useful? For Phase A/B t we suggest that total cost and 
total labor hours for each CBS is most useful for evalu- 
ation. The CBS should be correlated with the SOW so it 
is possible to relate cost and labor to each part of the 
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SOW. Total cost , time phased by month or fiscal year ; 
can also be provided. The SF 1411 does require time 
phasing. If the 1411 is not required then the Discovery 
program office can decide on the appropriate level of 
cost detail. 

If the Phase A study has been completed, provide the actual 
cost data in the same level of detail as requested for estimated 
costs, to the extent possible. 

No comment. 

B. Technical Definition (Phase B) Cost Estimate. This 
section provides a cost estimate for performing the Technical 
Definition (Phase B) study. Plans for the study should be 
described, but reference may be made to the Technical Ap- 
proach section of Volume 1 1 . 

If the Phase B study has been completed, provide the actual 
cost data in the same level of detail as requested for estimated 
costs, to the extent possible. 

I. Completing this section , the guidelines for Phase A 
apply except that the Contract Pricing Proposal is ONLY 
REQUIRED FOR THOSE INVESTIGATIONS PROPOSED 
TO BEGIN IN PHASE B. 

Same comments as for Phase A apply. 

C. Design/Development Phase (Phase C/D). This sec- 
tion provides a detailed cost proposal for performing Design/ 
Development Phase C/D. Plans for the Design/Development 
phase (Phase C/D) should be described, and a correlation of 
the costs with the technical approach should be included. 
Reference may be made to the Technical Approach section of 
Volume 11. 

This paragraph should just state that the costs should 
be correlated to the technical and management ap- 
proach defined elsewhere in the proposal. 

In completing this section, the following guidelines will 
apply: 

1. PhaseCID Cost Breakdown. ACostBreakdownStruc- 
ture (CBS) for every year of the Design/Development Phase 
(Phase C/D) must be included in the proposal. This CBS shall 
be to the subsystem level (level 3) for the flight system, and 
for all other cost items at least the system level (level 2). The 
value of all reserves, contributions, the cost of launch vehicles 
and services, and any facility and equipment costs shall also 
be included. 

Defining a level (level 2, level 3, etc.) causes us to 
create a CBS that is different than we usually use and 
makes it more difficult to relate to a SOW. We prefer that 
the instructions state something like: 


Costs and BOEs shall be provided: to the subsystem 
level (ACS, Propulsion, etc.) for the spacecraft and to 
the instrument level for the payload. All other costs and 
BOEs shall be provided at the major task level (program 
management, systems engineering, etc.). 

Regarding TDs and BOEs. 

The SOW should be the TDs. It is redundant and 
perhaps confusing to require a SOW and TDs. The SOW 
defines the work/products to be provided for each CBS 
cell. 

The BOE describes the methods and sources used to 
estimate the labor hours, cost, materials, and subcon- 
tracts at the required level. For hardware the BOE in- 
cludes an equipment list with the major components 
their sources and heritage and cost detail where appro- 
priate. The BOE needs enough detail so an evaluator 
can make a reasonable assessment of the validity of the 
cost proposal. Detailed material lists (resistors, wire, 
etc.) should not be required. A total materials cost should 
be adequate. 

The Design/Development phase should be summarized by 
major elements of cost for each cost category in the CBS. The 
elements of cost for the Phase C/D cost estimates should 
include the following, as a minimum. 

No comment. 

(a) Direct Labor. List by laborcategory, with laborhours 
and rates for each. This should correlate with the workforce 
staffing plan discussed below in Section 2. If Civil Servant 
labor is to be used, but is not to be charged directly to the 
investigation, then this labor must be considered as a contri- 
bution by a domestic partner, subject to the same restrictions 
as other contributions by domestic or foreign partners (i.e., 
the sum of such contributions should not exceed approxi- 
mately one-third of the Phase C/D development cost to NASA). 

See previous comments on labor categories and civil 
servants. 

(b) Materials. This should give the best estimate of the 
total cost of the bill of materials. Identify separately the 
estimated cost of major items, if known. 

Major items should be defined. We suggest that major 
items be defined as important components of a sub- 
system (gyros, a computer, fuel tank, etc.) or costing 
more than some amount, say $ 100K. 

(c) Subcontracts. List any major subcontracts (antici- 
pated and known), and the basis for estimated costs. 

Major subcontracts >$500K. 
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(d) Other Direct Costs. Include launch vehicles and 
services, facilities, and equipment. Any costs that are not 
covered elsewhere, including insurance, travel, etc., should 
be itemized here. 

Itemization should be at a summary level by major 
category. 

(e) Indirect Costs. This includes all overhead, general 
and administrative, fee, and any other miscellaneous ex- 
penses related to the overall business. 

No comment . 

2. Provide a preliminary workforce staffing plan. One 
that includes all management, technical (scientific and engi- 
neering), and support staff by fiscal year. 

Perhaps this overall staffing plan is enough data on 
labor classification that cost by labor category is not 
required in a. above . 

3. The cost estimate. It shall include all burdens and 
profit/fee in real-year dollars by fiscal year, assuming the 
inflation rates used by NASA (provided in Appendix E) or 
specifically identified industry forward-pricing rates. 

No comment 

4 . Provide a description of the cost-estimating model(s) 
and techniques used in your Phase CID cost estimate . Dis- 
cuss the heritage of the models applied to this estimate includ- 
ing any known differences between missions contained in the 
models and key attributes of the proposed mission. Include 
the assumptions used as the basis for the Phase C/D cost and 
identify those that are critical to cost sensitivity in the inves- 
tigation. Discuss the project risks that result from an uncer- 
tainty analysis of the cost estimate and provide the attendant 
total cost estimate range these risks create. Discuss the meth- 
odology by which all cost risks will be identified, tracked, and 
mitigated by the technical management process applied in 
this investigation. Identify any “discounts” assumed in the 
cost estimates for business practice initiatives or streamlined 
technical approaches. Describe how these have been incorpo- 
rated in the cost estimate and will be managed by the inves- 
tigation team. 

Suggest this paragraph be deleted. If a proposer uses 
models he/she can describe them in the BOEs. Risk 
analysis can be required as part of the technical and/or 
management sections . 

5. Provide a funding obligation plan. One for the pro- 
posed funding requirements of the investigation by annum 
keyed to the work schedule. 


No comment 

6. Provide a schedule for accomplishing Phase C/D ac- 
tivities. All funded schedule margins should be identified. 

Schedules should be part of the management section 
but could be duplicated in the cost section. 

7. Contract Pricing Proposal. (ONLY REQUIRED FOR 
THOSE INVESTIGATIONS PROPOSED TO BEGIN IN 
PHASE C/D). 

No comment. 

(a) The cost proposal. It will include, as a summary of 
total proposed Phase C/D costs, a completed SF 1411, as 
included in Appendix C. 

Only required if there is no competition. 

(b) The SF 141 1 must be signed by the proposer's autho- 
rized representative. 

No comment. 

D. Mission Operations Phase (Phase E) Cost Estimate. 
This section provides a cost estimate for performing the 
Mission Operations for Phase E. Reference may be made to 
the Technical Approach section of Volume II. In completing 
this section, the guidelines for Phase C/D apply. 

The C/D comments apply here. 

E. Total Mission Cost ( TMC ) Estimate. This section 
should summarize the estimated costs to be incurred in Phases 
A through E including the following: 

1. Preliminary Analysis Study , Phase A. 

2. Technical Definition , Phase B. 

3. Design and Development Phase , Phase CID. 

4. Mission Operations and Data Analysis Phase, Phase E. 

5. Launch vehicle , upper stages , and launch services. 

6. Mission-unique costs to the Deep Space Network and 
other ground system costs . 

7. Cost of activities associated with technology transfer 
and programs for social or educational benefits (if not incor- 
porated in any of Phases A through E). 

This section should include: Detailed plans for all as- 

pects of the mission not discussed elsewhere in this volume, 
including the launch vehicle, upper stages, and launch ser- 
vices; Deep Space Network and other ground systems; activi- 
ties associated with technology transfer and programs for 
social or educational benefits. Reference may be made to the 
Technical Approach section of Volume II. In completing this 
section, the following guidelines will apply: 
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No comment. 

1. Funding Profile Versus Time. A summary of the Total 
Mission Cost time-phased by fiscal year must be included in 
the format shown in Figure D2. Dollar amounts should be 
shown in real-year dollars. Total Mission Costs should be 
summarized in both real-year and FY92 dollars in the last two 
columns of this table. This summary should represent the 
optimum funding profile for the mission. Assets provided as 
contributions by international or other partners should be 
included, and clearly identified, as separate line items. 

No comment. 

2. Total Mission Cost Breakdown by Institutional 
Category. A summary of the total costs to NASA for the 
investigation, broken down by institutional categories (i.e., 
educational institutions, industry, nonprofit institutions, 
NASA Centers, and other Government agencies) and using 
the template in Figure D3, should be included. Participation 
by small, minority- or women-owned, and/or otherwise dis- 
advantaged businesses should also be highlighted, as should 
participation by historically black colleges and universities or 
by other minority institutions. Indicate the page(s) in the 


proposal where the participation of each institution is docu- 
mented. 

No comment. 

F. Tracking and Phasing of Schedule Margins and 
Cost Reserves. Specific margins and reserves in cost and 
schedule should be identified by phase and year and the 
rationale for them discussed. The specific means by which 
costs will be tracked and managed should be defined. Specific 
reserves and the timing of their application, if needed, should 
be described within the proposal. This should include the 
strategy for maintaining reserves as a function of cost-to- 
completion. All funded schedule margins should be identi- 
fied. The relationship between the use of such reserves, mar- 
gins, potential descope options, and their effect on cost, 
schedule, and performance should be fully discussed. 

This important requirement , which should also in- 
clude a fee plan, probably belongs in the management 
section as part of the management plan. This is sug- 
gested because the reserve and management plan should 
be evaluated by management experts rather than cost 
experts. 
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Workshop Presentations 


Science Community Perspectives 


William V. Boynton, University of Arizona — 

What Was “Right”? 

Selection was by the book 

❖ Politics had little to do with the selection 

♦ Exception: Getting Congress involved was deplorable 

❖ Weighting factors were followed 

Great care was taken to have a very rigorous review 
Debriefings were very valuable and informative 
AO was thorough and explicit 

What Was Wrong? 

Selection was by the book 

❖ Weighting factors were followed 

❖ Evaluation criteria were wrong (obviously subjective) 

Community is in the dark as to what was selected and why 

❖ Other than initial press release, I know of no description of the missions 

♦ Exception: Binder was liberal with passing out fact sheets 

❖ What are the science goals? 

♦ How are they going to achieve them? 

What is their education and outreach program like? 

Other nonproprietary information would be useful 

Perception = Reality? 

Don’t know 

❖ More information from NASA would help 

Science community support will be important for program to succeed (or continue) 

❖ Code SL uses scientists to satisfy NASA’s goals for the nation 

❖ Scientists use NASA to allow them to advance their scientific understanding and create 
new knowledge for students 

Openness at this point would allow the community to assess what is REALLY good and what is bad 
about the program and make suggestions for change 
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Science Community Attitude 

Originally very enthusiastic about the Discovery Program 

❖ As a way to do good science 

❖ As a way to get project management more responsive to science 

University as a lead institution was an important part of that 

❖ Peer-reviewed missions were thought to be a good idea 

Got mission selection out into the open 

Much of the science community is now soured on the Program 

❖ Rightly or wrongly 

❖ Feels that cost was given too much emphasis 

❖ Feels that many first-rate missions were passed over 

♦ Selected missions were not like those being recommended by committees 

♦ Selected missions were not “mainstream” but could have been 

Perceived / Real Problems 

Not a university-based program 

❖ Originally Discovery sold as a partnership with universities, industry, and government with 
universities as the lead 

❖ Evaluation made it difficult to run one of these from a university, even one with experience 
with large space projects 

♦ This was fair that the lesser experience of a university with big projects compared 

with, for example, JPL, be given some weight to compensate for the added risk 

♦ But there appeared to be no offsetting strength in having a university lead to balance 

♦ If NASA sees no advantage in the Discovery Program having more of a university 

component than past missions, it should state that explicitly 

Difficulty in getting a competent review panel 

❖ Most knowledgeable investigators were involved in proposals 

❖ Requirement to eliminate even the hint of a conflict kept good people off the panel 

♦ Scientists could probably review the quality of the science but would be unable 

to judge the likely success of the approach 

More Perceived /Real Problems 

No way to fold programmatics and long-range strategy into selection decision 

❖ Strategy means looking over the very long term (10-15 years) to formulate a program 
that gets the data necessary to answer the big questions 

❖ National Academy of Sciences sets the strategy 

❖ NASA is charged to implement the strategy 

❖ Science evaluation (presumably) was on basis of its intrinsic merit and not much weight 
given on how it helps fill in the strategy 
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Opportunity was a sham 

❖ Evaluation weights did not reflect the public rhetoric coming from NASA 

❖ Pluses for use of new technology did not balance minuses from risk 

❖ Education and public outreach were worth less than two percent of score 

More Perceived / Real Problems (Costs) 

Cost given too much emphasis 

❖ Formal weight was only slightly greater than science 

❖ Science was probably not ranked with the full dynamic range of cost (i.e., suspicion is that 
most science was good to excellent) 

❖ Future cost to implement the NASA strategy not considered (for example) 

♦ A $200M mission with “very good” science that addresses half of the first-order 

science objectives for a body 

♦ A $300M mission with “excellent” science that addresses all the objectives 

♦ If the former is selected, another $200M mission is needed for a total cost of $400M 

vs. only $300M to do the one complete mission 

More Problems with Cost Evaluation 

Evaluation of cost on the basis of what NASA says it will cost, rather than what the proposer says, 
seems strange for a cost-capped program 

❖ What is the award; what the proposer requests or what NASA thought it would take? 

♦ If the former, than it makes no sense to do the evaluation on the basis of the 

NASA-estimated cost 

♦ If the latter, it seems unreasonable for NASA to award more than the proposer asks 

for when its estimate is higher, and it seems unfair to award less, if its estimate 
is less than the proposer asked 

♦ But if the evaluation is based on a low NASA-determined cost, it seems unfair to the 

other proposers that the evaluation be based on one number and the award be 
given on the basis of another 

♦ If the policy is to award no more than what is asked, the evaluation should be on 

those costs and the appropriate risk assigned to success on the basis of the 
proposed cost 

What Should NASA Do? 

Decide what goals they want the program to accomplish for the nation 

❖ Is education and outreach really important? 

❖ Is technology transfer and infusion really important? 

❖ How important is the small and small/disadvantaged business involvement? 

❖ How important is university and student involvement? 

♦ Currently it is less important than minority involvement 

❖ Is it really business as usual with missions being managed by NASA centers? 

❖ Structure the program to reflect these goals 
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Make the words from NASA reflect the reality 
Publish details on the missions 

❖ From proposal: fact sheets, executive summary, science, opportunity, management 

Evaluate costs on the basis of proposed costs and the risk of the proposer being able to do it at the 
proposed cost. 



C. T. Russell, University of California, Los Angeles — 

Discovery Experience 
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San Juan Capistrano 

❖ PI — Venus Orbiter - IR/Lightning/Ions 

♦ Not selected 

❖ Co-I — Mercury Orbiter - Imaging/Lidar/Magnetosphere 

♦ Selected for pre-phase-A study 

Announcement of Opportunity 

❖ PI — Diana - SEP mission to Moon and Comet 

♦ Not selected 

❖ Co-I — Hermes - Mercury Orbiter 

♦ Not selected 

❖ IDS— VESAT - Venus Orbiter - IR 

♦ Not selected 


NEAR 

❖ Team member, magnetometer 


Science Advisory Experience 

Space and Earth Science Advisory Committee 
Space Science Board 

Committee on Data Management and Computation 
Planetary Science Data Steering Group 
Numerous ad hoc studies 
Participant in many planetary missions 
Apollo 1 5 
Apollo 16 

Pioneer Venus Orbiter 

Vega 1,2 

Phobos 

Galileo 

Mars 96 

Cassini 

NEAR 
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The First Two Discovery Missions 

Mars Pathfinder 

❖ Seemingly arbitrarily chosen 

❖ Principally an engineering mission 

❖ Strong non-U.S. science contribution 

❖ JPL appears to be managing it well 

❖ Dead-end project 


NEAR 

❖ Had strong science support 

❖ No apparent competition for spacecraft 

❖ No apparent competition for mission operations 

❖ Science team brought on late in project 

❖ APL is doing a good job 

❖ Should be a good first asteroid mission 

San Juan Capistrano 

The Numbers 

❖ 73 Submissions 

❖ 4 Selection panels 

❖ 2 Ratings 

The Process 

❖ Oral presentation 

❖ Evaluation of 10-page white paper 

❖ Rapid decision on 1 0+ winners 


The Result 

❖ Surprise 

❖ Uneven assessments by panels 

❖ No coherence to selected program 

❖ Obvious holes - lunar 

❖ 1 0+ underfunded teams 

Pre-Phase-A Study 

❖ Too little money 

❖ Too long a period 

❖ Much thought and discussion 

❖ Ideas and designs matured 

❖ Seemed principally a holding pattern until the real program began 

❖ Most progress when AO was imminent 
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The Draft AO 

❖ Released in early 1 994 

❖ Helped guide final directions of pre-phase-A studies 

❖ Enabled problems in instructions to be removed, reducing cost of proposal preparation 
and providing better balance in technical vs. cost sections of the proposal 

❖ The resulting AO was good document 

❖ The delay in the release of the AO was disconcerting 

❖ The lowering of the cost cap after the AO was released was inexcusable 

Selection Criteria in AO 

❖ “Cost and Management” will have approximately the same importance as “Science, 
Technical Approach, and Opportunity” 

❖ Science will be rated at approximately the same weight as the combination of Technical 
Approach and Opportunity 

❖ Technical Approach will be weighted significantly more than opportunity 

❖ Cost will be more important than Management 

Lessons Learned 

Science Objectives 

❖ Neither NASA’s nor the Academy’s planetary exploration strategy was factored into 
selection. 

❖ Quality of science was assessed but not the quantity. For example, two bodies completely 
explored were no more important than one body explored. 

❖ Science ratings were quite different than we were used to experiencing. 

Past Rankings Category Result 

Excellent 1 May receive funding 

Very Good 2 Consolation prize 

Good 4 Kiss of death 

Discovery Rankings 

Excellent ok 

Very Good ok 

Good ok 

Lessons Learned 

Evaluation Process 

❖ Science evaluation took place week of fall AGU meeting. The week most active planetary 
scientists are fully committed. 

❖ Potential first-rate evaluators had to decline to participate due to prior commitments. 

❖ Errors in interpretation and misunderstandings of the proposals occurred but no questions 
were asked. One of these misunderstandings was thought to be fatal for Diana! 

❖ The one question that I was asked was addressed clearly in the proposal. 

❖ It was clear early that the evaluation process was flawed. 



20 Workshop on Discovery Lessons-Learned 


Lessons Learned 

Costing 

❖ Costing is a very inexact science. 

❖ Industry and NASA Centers are not very frank with each other about costs. 

❖ Unexpected and unexplained costs were added by JPL at the last minute. 

❖ Cost models were used to evaluate proposals that may have favored one set of approaches 
over another unfairly. 

❖ Cost models are not available for SEP. 

❖ Different proposers took different approaches to determining costs. 

Lessons Learned 

New Technologies 

❖ Cost risk. 

❖ Schedule risk. 

❖ Cost models overprice new technology, especially SEP. 

❖ Use of new technology is a net negative in proposal evaluation, no matter how important 
the introduction of that new technology. 

❖ Solar Electric Propulsion would open up a whole new era of planetary exploration at 
reduced costs, yet it might as well have been the introduction of a new washer as far as 
evaluation was concerned. 

❖ Use of old technology was rewarded. 

Lessons Learned 

Expense 

❖ Discovery proposal preparation can be very expensive. 

❖ Perhaps $0.5M was spent by many of the teams. 

❖ Very good pre-phase-A studies resulted. 

❖ Since selection did not seem to reflect the depth of the pre-phase-A study, much effort 
seems to have been wasted. 

If only “high-level” concepts desired then AO should say so. 

*> Community cannot afford the expense required to prepare Discovery proposals at this 
level on a continuing basis. 

Lessons Learned 

Partnerships 

❖ Discovery mode necessitated partnerships: industry, centers, universities. 

❖ Partnerships require meaningful sharing of responsibilities — ownership of part of the 
effort. 

❖ Meaningful sharing of responsibility means complex organization charts. 

❖ Complex management with checks and balances is viewed as a negative. 

❖ A meaningful partnership with split responsibilities was a negative in the evaluation 
process. 
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Lessons Learned 

Science Centers: Friend or Foe? 

❖ Goddard and JPL have scientific as well as engineering efforts. 

❖ These centers look out for their scientists. 

♦ do not work on competing efforts 

♦ refuse to assist unless center scientists involved 

❖ These centers do not provide partial assistance, e.g., operations but not program 
management. 


Conclusions 

❖ Many mistakes appear to have been made in the evaluation process this time. 

❖ Even if there were no mistakes, many questions whether in the presently constrained 
environment whether we can afford to fund merely good science. 

❖ We need to leam how to correctly cost modem spacecraft, built in competitive environ- 
ments. Too much of our cost experience is based on situations where funding was not 
capped and there was no competition. 

❖ We need to leam how to evaluate science return quantitatively so that the science per 
dollar is known and factored into the evaluation process. 

❖ Excellent selections will lead to a strong, continuing Discovery program. 

❖ Merely good selection will lead to dissatisfaction with the program and its eventual 
demise. 
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Industry Perspectives 


D. Roalstad , Ball Aerospace — 

THE DISCOVERY PROCESS IS INNOVATIVE AND PROGRESSIVE 
Improvements Experienced 

IPDT is a winning ingredient formulation process. 

❖ No force fits (some exceptions) 

❖ Avoidance of intermediary functions 

❖ Avoidance of institutional demands 

❖ Smallest possible team 

PI leadership for high science return 

❖ Direct control of trades to maximize science 

❖ Responsibility with authority 

Manageable Programmatic Guidelines 

❖ Cost and schedule (risk, technology, science, trades) 

❖ Contingency management (PI responsibility) 

Improved Procurement Process 

❖ Demonstrated so far 

❖ Improvements possible from recommendations 
Progress has been made — “Lessons Learned will refine” 


Cost — A Primary Issue of Concern 

Can science return per dollar be determined? 

Should cost outweigh science for AO selection? 

Is AO proposal detail cost information meaningful? 

Are innovative ideas squelched with early cost emphasis? 
Should AO proposal cost information be eliminated? 


Cost discriminators should be based only on well-studied concepts and engineering details. 
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B. C. Clark, Lockheed/Martin — 

Discovery Missions — The Concept 

A New Way of Thinking 

❖ Mind-stretching 

❖ Lag time in generating concepts 

❖ Are we all in sync yet? 

Architecture Fostered New Competitive Approaches 

❖ Academia/industry/govemment forced on equal footing 

❖ Emphasi s on minimization of cost and complexity 

Revolutionary Jump Beyond Performance-Based Specs 

❖ Team defines the goals, program, and solutions 

❖ Teams formed around program’s needs (altruistic division of work tasks) 

Essential to the Future of Planetary Science 

❖ Multiple opportunities hold the scientists in the field 

❖ Multiple opportunities foster industry interest and investment 

What We Did 

Outstanding Set of Worthy Missions 

❖ Existence proof of excellent science at affordable levels 

Multiple Potential Winners Led to Distributed Effort 

❖ Major involvement in 8 missions (7 spacecraft, 1 integrated payload) 

❖ Minor involvement in 2 other missions 

AO Response was a Major Effort 

❖ Not the typical AO; much more like an RFP and not unexpected 

❖ Standardization of responses was not as feasible as hoped 

❖ Work effort was excessive, but irrevocably locked into several proposals 

Delays in AO Issuance was Major Perturbation 

❖ Resulted in additional expense 

❖ Difficult to keep team on track 

Smaller, Less Complex is Indeed Cheaper 

❖ Avoid big-program syndromes 

(exponentiation of communications and conservatisms) 
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Principle Investigator as Leader 

Wide Range of PI Characteristics 

❖ Ranged from instrument/experiment specialists to theoreticians 

❖ Ranged from hands-on to hands-off approaches to management 

❖ How can/did the evaluation team consider these variations? 

Relationships of PI, Project Managers, and S/C Managers 

❖ Academia/industry/govemment in novel relationships 

❖ Stimulated revolutionary thinking 

Good-PI Dilemma 

❖ Each felt their science objective was sure winner 

❖ Highly competent Pis already very busy; some also have talent for management 

❖ Pis not always comfortable with highly structured schedules (science sets its own pace) 

Team Building 

❖ Scientists became engineers; engineers became scientists 

❖ Badgeless, altruistic attitude developed 

The Evaluation 

Thoroughness Matched the Effort 

❖ Large, systematic team evaluation commensurate with large, detailed proposals 

Debriefs were Outstanding 

❖ Thorough, specific, definitive 

❖ Extremely valuable for assisting future actions 

Focused Science 

❖ The 1 to 3 instruments concept (San Juan Capistrano) not propagated to the AO 

❖ Unclear if highly -focused science was favorable as expected . . . some downchecks 
from scientists (Catch-22 situation) 

The Future 

Evaluating the Science/Cost Ratio 

❖ Cost is easy 

❖ Science vs. science is more difficult to evaluate numerically 

♦ Begin with rank-ordering 

♦ Use adjectival descriptors (is the investigation essential? amount/fidelity of information? 

breadth of applicability?) 

♦ Pivotal to assign as large a point spread as possible 
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Should There Be a “Small Mission Set-Aside?” 

❖ Balance between cheapest missions and more rewarding/more expensive missions 

❖ Don’t throw the baby out with the wash 

Do Mars or Outer Planet or Asteroid Missions Have a Chance? 

❖ Mars Surveyor Program 

❖ Pluto Flyby Program 

❖ NEAR Mission 

❖ New Millennium Program 

Relationships to Other Programs 

❖ Mars Surveyor Program 

❖ New Millennium Program 

❖ Overlaps of mission goals/targets 

Additional Recommendations 

Consider a Multistep Process 

❖ Step 1: 

Screening to reduce the field to a reasonable number of semi-finalists (8?) 

25 total pages, combined science/ technical/outreach/management/cost 
(defer Outreach to Step 2) 

45 days to prepare and submit 
45 days to evaluate and down-select 

❖ Step 2: 

Proposal using the ’94 AO level of detail 
60 days to prepare and submit 
60 days to evaluate and down-select 

❖ Steps 3 and 4: 

Proceed as currently planned (run-off between two or three finalists) 

Cost Categorization and Definitization for Step 1 

❖ Example: Nearest $25M increment estimated cost 

❖ Clarify criteria: Total mission cost vs. peak funding vs. C/D funding 


Cost Floor 

❖ Hitting a barrier for several highly interesting and important missions 

❖ Law of diminishing returns 
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J. Freitag, TRW Space and Electronics Group — 

Discovery Program Represents a New Way of Doing Business 

The Discovery Program was characterized by new ideas 

❖ Fly a mission every 1 2 to 18 months 

❖ Cap costs at $150M 

❖ PI led partnerships with industry and government centers 

Overwhelming response from industry, academia, and government centers 

❖ San Juan Capistrano 73 Submissions 

❖ NASA funded pre-phase-A studies 1 1 Missions 

❖ Response to AO 28 Responses 

❖ Chosen for implementation 1 

❖ Chosen for phase A studies 3 

It is encouraging that NASA has organized this workshop 

❖ New ideas produced overwhelming response — and selection shock 

❖ Without revisions, will the next AO receive the same response? Has it impacted Midex 
and New Millennium? 

Discovery is one of several “new ways” of doing business with NASA 

❖ Can these other “new ways” help refine the Discovery process? 

Workshop Charter: Develop recommendations for the next AO 

How Can TRW Contribute to the Goals of this Workshop? 

Reaffirm Support and Enthusiasm for the Discovery Program 

Encourage implementation of selected missions 

❖ Allow selected teams to demonstrate their proposed competence 

❖ Support notion that teams live up to their proposed responsibility 

❖ Terminate when performance does not meet proposed performance 

Identify issues for splinter group discussion 

❖ Science and Return on Investment 

❖ Technology vs. Risk: Mission Success; and other factors 

❖ Cost 

❖ AO/AO Process Changes and Program Management 
Science and Return on Investment Issues 

❖ Will current emphasis on low cost and risk reduction encourage conservative science 
for the next round? 

❖ Should proposed Discovery Missions relate to NASA-defined solar system objectives 
and be evaluated according to their proposed achievement? 

❖ Should the metrics of science return on investment be applied to all Code S-type 
programs? 
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Technology vs. Risk: Mission Success; and Other Factor Issues 

❖ Were programs selected on their ability to propose acceptable risk, or the conservatism 
of the evaluation process? 

❖ What is the impact on future AO responses when higher-risk missions are penalized during 
the evaluation process? 

❖ If evaluators find a risk is unacceptable, should proposers be allowed to present additional 
data to show the risk is tolerable or there is an adequate risk mitigation? 

Cost Issues 

❖ Given the heavy weighting of cost during the evaluation, does a cost cap make sense or 
should a target cost be specified? 

❖ Should the proposal/evaluation process be modified to allow the proposers to demonstrate 
cost credibility through new ways of doing business? 

❖ Are current cost models adequate for evaluating 28 diverse mission proposals? 

❖ Can incentives and penalties be built into the process that encourage more realistic 
proposals? 

AO/ AO Process Changes and Program Management Issues 

❖ Are the length of the Discovery process and the investment resources required to support 
the process excessive? 

❖ Would completion of science evaluation prior to partnership proposal preparation process 
reduce program process time and investment? 

❖ Given the 24 proposals that were not winners, what is the basis for resubmitting any of 
these again? Would the same evaluation process be used in the next round? 

❖ How does the Discovery “new way” of doing business relate to other NASA “new ways” 
of doing business? 

❖ Will the New Millennium Program impact the AO content for the next round of 
Discovery? 

Summary Comments 

Discovery must do more science for less money 

❖ Current climate: less money for science 

❖ More knowledge is always better 

❖ But what is the limit? 

American needs a sense of priorities to guide diminished spending 

❖ Federal R&D budget decline leads to industry budget decline 

♦> Balancing federal budget and corporate efficiency must not jeopardize future 

Can metrics be applied to adequacy of R&D spending 

❖ American competitiveness in solar exploration 

❖ Value of result vs. investment 

Contributions from young scientists to maintain continuity of talent and progress 
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D. Gump , LunaCorp — 

LunaCorp 

❖ Founded in 1 989 to find private funding for space exploration 

❖ Formed Lunar Eclipse Software subsidiary in 1993 

♦ Authored/published Return to the Moon CD-ROM in 1993 

♦ Authored/published Mission: Planet Earth CD-ROM in 1994 

❖ 1 998 Lunar Rover Mission 

♦ Will land two teleoperated rovers on the lunar surface 

♦ Red Whittaker of Carnegie Mellon University Robotics Institute is designing and 
building the rovers 

♦ Mission cost is $150 million 

♦ Revenue from entertainment, television, corporate, and research customers 

♦ Public participation is key component of mission 

♦Mission Control will be at theme park 

♦Theme park visitors will drive the rovers live and explore via telepresence 
Data Purchase as Discovery Option 

❖ Discovery rules should allow the purchase of science data from commercial missions as 
alternative to NASA-managed projects 

❖ Rules excluded LunaCorp in last round 

❖ LunaCorp has offered NASA lunar surface data for low cost 

♦ $300,000 per payload pound 

♦ $7,000 per dedicated rover time 

❖ Data purchase is only strategy likely to secure New Start funding from Congress 
Data Purchase Contracts 

❖ Data purchase contracts should have same progress payments and completion penalties as 
NASA-managed projects 

♦ Majority of contract money is paid out prior to launch 

♦ Launching the mission satisfies most contractor requirements 

♦ Successful return of data releases the final 1 0% of the contracted amount 

❖ To protect government, the data seller must post bonds to guarantee repayment if mission 
doesn’t launch. This should constitute the only financial test the company must satisfy. 

❖ In comparing proposals, the costs should be all-inclusive: 

♦ Including cost of launch vehicle and launch site/range 

♦ Including estimates of the government’s self-insurance costs 

♦ Including mission operating costs 
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THE LUNACORP TEAM 

Thomas F. Rogers, chairman of the board. As Pentagon’s Deputy Director of Defense Research and 
Engineering, he was responsible for the general design and deployment of the first global 
satellite communications sytem. 

David Gump, president. Mr. Gump is founding publisher of Space Business News, author of Space 
Enterprise: Beyond NASA, and former marketer for Geostar Messaging Corporation. 

Victoria Beckner, public relations and marketing manager. Ms. Beckner is a former political con- 
sultant, and as a NASA contractor she headed up public relations for the Microgravity Sci- 
ence and Applications Division. She is the founder and former editor of Microgravity News. 

■ James Dunstan, executive vice president. Mr. Dunstan is a partner at Haley, Bader & Potts, an 
Arlington law firm active in communications and space law. 

Rick Thmlinson, director. President of the Space Frontier Foundation in New York. 

Philip Culbertson, advisor. Mr. Culbertson is a former general manager of NASA. 

Walt Anderson, director. Founder of Mid Atlantic. He is now chairman of Esprit Telecom, the first 
pan-European carrier in the newly deregulated communications market in Europe. 

Dr. William C. Stone, director. Dr. Stone is the developer of the MK-2R backpack, a computer- 
controlled diving rebreather. 

Scott Carpenter, advisor. During the Mercury program, Scott Carpenter was the second American 
to reach orbit, piloting his Aurora 7 capsule in May 1962. 

Paul J. Coleman, Ph.D., advisor. Professor of space physics at UCLA, and President of the Univer- 
sities Space Research Association, a 76-university consortium. 

Patrick Quentin Collins, Ph.D., advisor. Former consultant at the European Space Agency’s 

Research and Technology Centre, he has taught at Imperial College in London since 1 983 and 
is currently a visiting professor in Japan. 

Allan S. Hill, advisor. At Boeing and Northrup Space Laboratories, Mr. Hill designed and developed 
the Burner II and IIA Thor and Atlas upper stages and the Saturn S-IC booster stage. 

Edward J. Martin, advisor. During his fifteen years at the Communications Satellite Corp. 

(COMSAT), Mr. Martin’s posts included Vice President, Technology Management, and Vice 
President, International Operations. 

George E. Mueller, Ph.D., advisor. Dr. Mueller is President of the International Academy of Astro- 
nautics. He was NASA’s Associate Administrator for Manned Spaceflight from the start of 
the Gemini program through the second Apollo landing. 


For more reference, see Popular Science, June 1994 issue, and Newsweek, December 5, 1994 issue. 
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Government /FFRDC Perspectives 


G. Scott Hubbard, NASA Ames Research Center — 

AMES CENTER PERSPECTIVE 

Summary 

Discovery is an excellent Program (a number of people deserve credit for making it happen) 

Don’t make drastic changes 
Fine tune a few areas 

Ames Support for Discovery: Past, Present and Future 

Past: 

Planetary science and project development at Ames consistent with Discovery 

❖ Small/medium projects (Pioneers 6-13, Galileo probe), numerous Pis and instruments 

❖ Development approach has been teaming with industry 

❖ Minimize requirements growth, be cost effective, utilize small management oversight team 

Participated in first Discovery Science Working Group in December 1989; all subsequent SWGs 
and workshops 

Present: 

Involvement in current Discovery project development 

❖ Developed Mars Pathfinder (MESUR) mission concept at Ames in 1 990 

❖ Collaboration with JPL on Pathfinder en try/ descent/landing system and instrumentation 

Discovery AO selections — Ames-related involvement 

❖ Lunar Prospector: NASA Center and Co-Investigator 

❖ Venus Multiprobe Mission and Stardust: Co-Investigators 

Future! Summary: 

Discovery represents a unique opportunity for planetary missions 

❖ Open competition peer review is the best mechanism for selection 

❖ Process is complementary to proposed Ames Institute 

Discovery AO General Comments 

AO accommodated a wide range of proposals 

❖ Diverse scientific and technical interests should be maintained 

Overall, the AO was clear and provided the background and guidelines necessary to write 
the proposals 

❖ Some repetition could be eliminated and details improved 
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The importance of science/cost ratio was not fully appreciated by all proposers 

❖ Prior instrument AOs were based primarily on science excellence 

❖ Future AOs should maintain the emphasis on cost containment 

Proposals required significant resource commitment 

❖ Necessary aspect to yield high-fidelity cost data, but may limit participation 


Discovery AO Detailed Comments 

Rapid procurement process was enabled by Discovery process 

❖ Certification (SF- 1 41 1 ) of cost data was key to rapid contract award 

❖ Funding level uncertainty and delay in funding arrival at center slowed process 

❖ For consistent interpretation of “selection statement” might seek statement from 
HQ procurement 

Some budgetary terms were confusing 

❖ For example, cost vs. price, “contributed cost” vs. “actual” dollars 

❖ Provide more clarification in future AO 

Small Business and Small Disadvantaged Business goals were very challenging 

❖ Could lead to significant risk in management complexity; performance should be reviewed 
as program progresses 

Program Management Plan was not approved prior to the AO release 

❖ The draft plan should be finalized and published 

The requirements to incorporate new technology and minimize risk are contradictory 

❖ Need further discussion of the role of new technology 

Provide cost requirements in a more easily understandable format 

❖ Mixture of FY92, real year and current year costs were confusing 

Ames and Discovery: Summary 

Overall Discovery AO process was very positive 

Discovery AO allowed a broad range of proposals 

❖ Maintain this aspect 

Continue to emphasize cost-effective science 

❖ Science/$$ is the bottom line for healthy future 


Fine tune process in certain areas 
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E. Davis and E. Kieckhefer, Jet Propulsion Laboratory — 


Observations, Thoughts, Issues 

❖ Generally the Discovery concept is valid and is working 

❖ Improvements can be made in the process, some significant — focus on these 

❖ Discovery was a huge paradigm shift with significant and steep learning curve — but now 
we know, and it will be easier the next time 

❖ Overall, the amount of effort and/or investment required to get to the final outcome was 
too high 

♦ Significant NASA, industry, and JPL investments 

❖ Continue to form partnerships early — PI, IP, and JPL 

♦ Wait until final AO to form proposal teams 

♦ Use draft AO period to check the science, cost, and players to confirm proposal 

concept validity 

❖ Don’t change goal posts in the middle of the process 

❖ Focus on the AO; eliminate other up-front effort 

❖ Have a draft AO with final AO 60 days later and allow 90 days for proposal preparation; 
stick to the schedule 

❖ Simplify the AO and the proposal outline and contents — provide a very clear concise 
definition of the required proposal contents and only ask for information once and only 
that needed for evaluation 

❖ Select the next Discovery Mission starts (#5 and #6) directly from the proposals; don’t do 
a two-phase selection or down-select 

❖ The Discovery process with partnering/teaming and NASA HQ selection greatly 
streamlines the procurement process 

❖ Continue and improve the JPL Locomo thrust and workshop 

❖ Continue dialog with industry to improve partnering and teaming arrangements 

❖ Find a better way to link the Discovery Program’s outcomes with Code SL’s fundamental 
science goals 

❖ The evaluation outcome did not meet expectations with regard to technology 
infusion/application 

❖ The phase A funding shortfall was a rude surprise; don’t do this again 

❖ Must support the selected missions’ negotiated funding profile 
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L. J. Crawford, E. L. Reynolds, and R. W. Farquhar, Applied Physics 
Lab, Johns Hopkins University — 


JHU/APL Proposal Involvement 

❖ COmet Nucleus TOUR (CONTOUR) 

❖ Near Earth Asteroid Returned Sample (NEARS) 

❖ REndezvous with a COmet Nucleus (RECON) 

Presentation Outline 

❖ Announcement of Opportunity 

❖ Evaluation Process 

❖ Debriefing Process 

❖ Technology vs. Risk 

❖ Evaluation Criteria 

❖ Science vs. Cost 

❖ Program Management 

NASA’s Discovery Program Selection Process 



Good Schedule 


Major Shortcomings 

Concept of “Highest Science Per Unit Cost” is fundamentally flawed. 

❖ Cannot establish minimum “science floor” 

Drives community to “lowest common denominator” science 

Reliance on model to provide government-generated costs was unfortunate. 

Model over estimate costs 

❖ Models do not reflect new ways of doing business 

Use of industrial contractor (SAIC) in evaluation process was inappropriate. 

Minimal use of contractor’s past performance in source selection not in step with current 
government trends. 

❖ Minimum weight of 25% for past performance is new federal guidance 
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Alternative Discovery Selection Process 

Establish two or three classes of Discovery Missions, then don’t compete on basis of cost (will max 
science for each category) 

Launch Vehicle 

1. Delta-2 

2. Med-Lite 

3. Pegasus 

Compete missions within each class. 

Science value is primary factor in selection. 

Discovery Program would include some mix of mission classes [content of mix: TBD]. 

Announcement of Opportunity 

Too much information was requested (e.g., cost) 

Not enough page allotment for amount of information requested 
Next AO should contain numerical weights for each selection criteria 
Use only “one year dollars,” not several, e.g., 92, 94, real year 

Evaluation Process 

❖ Evaluation process was defined and describable. 

❖ Scatter-chart approach instantly removed missions viewed as moderately expensive and 
very expensive, even though they had excellent science and remained under the $150 M 
cost cap. 

❖ Aggressive missions with substantial science return were penalized because they were 
compared with simpler, cheaper missions primarily on the basis of cost. 

❖ Use of industrial contractors like SAIC in evaluation process is inappropriate. 

❖ Use of contractor-demonstrated performance in evaluating ability to deliver (cost, 
schedule, technical performance). 

❖ Reasonableness of contractor cost estimates. 

♦Don’t just use government cost estimates derived from models (models are known to 
overestimate). 

♦New guidance issued by the Office of Federal Procurement Policy (OFPP) recommends 
using past performance (weighted at a minimum of 25%) for source selection in 
government contracts [ref. Federal Contracts Report 0014-9063/95, 5/8/95]. 

❖ Process was very much business as usual and did not reflect new ways of doing business. 
♦Hard to see that innovation was rewarded. 

Debriefing Process 

❖ Debrief sessions were very useful, informative, open, well executed. 

❖ State the weighting factors for each category. 


C/D Cost Range ($FY92) 
100-150 
50-100 
<50 
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Technology vs. Risk 

❖ Proposals were both rewarded and penalized for new technology insertion. 

❖ Discovery originally used verbiage like acceptable risk. 

❖ NASA obviously wanted low cost, low risk; science no longer a primary driver. 

Evaluation Criteria 

❖ Science value should be increased above 25%. 

❖ Vol. II and Vol. HI proposal scores should be developed independently if similar approach 
is used in future. 

♦The technical, cost, and management evaluation team helped to determine both Vol. II 
and Vol. IE scores!! 



Science vs. Cost 

❖ Concept of “highest science per unit cost” fundamentally flawed. Drives community to 
“lowest common denominator” science. 

❖ Cost should not be the overriding factor for mission selection. 

♦ Establish whether program is within cost cap and then ignore costs as a selection 
criterion. 

❖ Science should override all other factors if costs are within caps. 

❖ Establish classes within Discovery so as to not penalize aggressive missions. 
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Program Management 

❖ Cost models. 

♦ Models used for evaluation should be given to all proposers. 

♦ Cost data should have no page limit. 

♦ Since models were used to determine cost, why was so much cost detail required? 

❖ NASA may have established a new cost cap for Discovery by accepting a $59M program. 

♦ If left unchanged, a void of science missions that exceed the $100M cap would 
be created. 

❖ NASA allowed too much visibility of the selection process — consequence was that other 
criteria could not be factored into process. 
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Summarized Written Comm u nity Comments 


Mark Saunders, Discovery Program Manager, NASA Headquarters — 


Community Response 

❖ Written comments were limited: 

♦ 2/3 of comments were submitted by Evaluation Panel membership 

♦ 1/3 submitted by members of Discovery missions 

❖ 2/3 substantial verbal comments during debriefings; not covered here 

❖ 1/3 comments came in four varieties: compliments, criticisms, suggestions for improve- 
ments, and questions 

Compliments 

❖ Discovery approach continues to be positive and AO implements Discovery 
goals/approach 

❖ Teaming approach with PI in charge is efficient and effective 

❖ Shifting risk management to teams puts mission assurance where it belongs 

Criticisms and Suggestions for Improvement 
AO 

❖ AO needs refinement to be more effective; too excessive for this stage 

♦ Responses cost too much 

❖ Conduct some form of two-step process that selects a small number of proposers from a 
summary proposal; request additional info from selected group 

❖ Mission cost should not be the overriding factor for mission selection; the best science 
within the “cap” should override all other factors 

❖ Page limits penalized aggressive missions since more information was required to 
describe missions 

Science 

❖ It appears that the central guiding principle of Discovery getting the most science for the 
dollar was not utilized in the evaluation process 

♦ A credible means of computing science per dollar is needed 

❖ A minimum threshold for science, as well as other criteria, should be established 

❖ The science evaluation should be based on the established Solar System Exploration 
priorities 

❖ Science criteria weight was too low; should have been at least 50% 

❖ Some policy for extended missions is needed 
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Technical Approach 

❖ Considering technical risk in both the technical approach and cost criteria is 
double jeopardy 

❖ Instruments should be considered as part of the technical approach 

❖ New technology should not have been penalized in technical approach, since phases A and 
B are there to deal with these issues 


Opportunity 

❖ SB/SDB participation, as well as other opportunities, may be too immature to include in 
the AO 


Management 

❖ Organizational experience was not considered in the evaluation of proposals 


Cost 

❖ AO required costing detail, which is inconsistent with stage of definition 

❖ Complete and accurate cost evaluation of 28 proposals was an impossible task 

♦ Sufficient documentation typically takes thousands of pages 

♦ Cost-estimating techniques are not sufficient discriminators 

♦ Cost model used in evaluation is inaccurate and not suitable 

❖ Exclusion of organizational cost performance was a fundamental flaw 

❖ Setting a cost target, as the mid-point, as well as the cost cap might help 

❖ Provide incentive/penalties to encourage accurate proposals 


Questions 

❖ How can more challenging scientific objectives be rewarded instead of penalized? 

❖ How should the 24 missions not selected consider their chances in the next round? 

♦ Some feedback from NASA would be helpful 

❖ How does Discovery relate to other NASA programs, e.g.. New Millennium, PIDDP, etc.? 

❖ How does new instrument development play? 

*:♦ Should cost even be a criterion? 

❖ A number of questions were asked about the evaluation process and results 
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NASA Headquarters Perspectives 


Mark Saunders , Discovery Program Manager, NASA Headquarters — 

General 

❖ AO, evaluation methodology, and evaluation process worked, but can be improved 

❖ Resource investments by both proposers and evaluators were extensive 

♦ Some other process needs to be explored to reduce investments 

❖ Execution of contracts was quick, but can be improved by requiring statement of work as 
part of proposal 

❖ In current environment, there are some program constraints that are nonnegotiable 

♦ Method that allows big and small missions to compete against each other is required 

♦ Assuring adequate cost estimating protects both NASA and proposers 

AO 

❖ Despite statements in AO and repeated attempts to advise community about importance of 
low cost, some never appreciated its significance 

❖ Five evaluation criteria worked well, but could be assembled differently: 

♦ Science and Return on Investment (ROI) 

♦ Probability of achieving scientific objectives 

♦ Technical Approach 

♦ Management Approach 

♦ Experience 

♦ Likelihood of making it within cost and schedule 

♦ Other factors, e.g., opportunity 

❖ Identifying weights of criteria would have helped proposers 

❖ Providing funding profile would have helped proposers 

❖ Since we plan to release AOs every 1 8-24 months, we should limit launch window or 
provide 2 flight opportunities with specific windows for each 

Evaluation Methodology 

Science 

❖ Prioritizing intrinsic merit of various targets/objectives would have helped both sides 

❖ Simple evaluation method for science, when combined with other criteria, may not have 
completely represented spectrum of science quality 

❖ Mail reviews would have been nice but would have lengthened process 

❖ Having cost weighed slightly more than science achieved our goal of making small, mod- 
est science missions competitive with more expensive/extensive science missions, but there 
may be better algorithm for science ROI 
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Technical Approach 

❖ Assessment of risk is valid, and is necessary constituent of technical approach 

♦ May be better way of balancing risk vs. technology infusion and scientific scope 

♦ Dependent on overall acquisition strategy that may differ from AO to AO 

❖ Some info was more important than others; would help to reduce data to that which is 
most important 

Opportunity 

❖ Too much may have been expected at this stage in areas of technology transfer, education, 
public affairs, and small/small-disadvantaged businesses 

❖ Most important is degree of commitment to each in terms of organizational and 
financial resources 

❖ Technology infusion might have more emphasis in future solicitations 
Management 

❖ Worked well, but certain areas could be adjusted to help proposers, e.g.: 

♦ Considering key position qualifications as well as key personnel experience 

❖ Some info was more important than others; would help to reduce data to that which is 
most important 

Cost 

❖ Absolutely most difficult area to evaluate; struggled with both techniques and consistency 
with program objectives 

❖ Conducted multitude of sensitivity analyses to confirm conclusions 

❖ Despite assertions to contrary, cost analysis techniques used did not skew conclusions 

❖ Another way to assess cost that assures fairness to all proposers and NASA is welcome 

♦ “Trust Me” (not assessing cost) will not serve either side well 

Evaluation Process 

Diverse science and technical teams provided sufficient viewpoints to assure fairness across all 
proposals and consideration of new ways of doing business 

Executive committee participation from start to end was beneficial and reduced overall labor and time 
Revising evaluation flow and better allocating resources would improve evaluation process 
A two-step process, similar to MIDEX AO, is possible while still achieving Discovery objectives 
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Two-step Process — one option 
Step #/: 

1 . Submit first half of proposal, which encompasses science proposal including mission 
design; include one-page summary of proposed cost; include basic management organiza- 
tion and team membership 

2. Evaluate science and ROI; select subset to proceed to step #2 
Step #2: 

1 . Submit second half of proposal, which includes technical approach; management and cost 
sections plus any changes to science 

2. Evaluate step #2 proposals based on three criteria previously described: ( 1 ) Science and 
ROI; (2) Probability of achieving scientific objectives; (3) Other factors (Opportunity) 

Note: Other factors ( Opportunity ) could be included in either step depending on emphasis 

Timeline: Step #1 proposal preparation 3 months 

Step #1 proposal evaluation 2 months 

Step #2 proposal preparation 2 months 

Step #2 proposal evaluation and selection 3 months 
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Splinter Group Sessions 


Science and Return on Investment (ROI) 
— W. V. Boynton 


Return on Investment should be determined by the following formula: 

ROI = 0.8 x Sci x SciRisk + 0.1 x Ed + 0.1 x PA 


Science Value (Sci) 

Science value is to be determined primarily by how well the proposed science objectives address the 
priorities determined by the strategy of the National Research Council (NRC) Committee 
on Planetary and Lunar Exploration (COMPLEX) and NASA’s strategic plans. The score 
should reflect the breadth of the proposed investigation (how many objectives are ad- 
dressed), the depth of the investigation (how thoroughly the objectives are addressed), the 
importance of the investigation (the priority of the objectives that are addressed), and the 
quality to which the objectives are addressed (the extent to which the proposed measure- 
ments are sufficient to address the objectives). 

Another more subjective factor to include is the anticipated impact that the investigation will have on 
the field. It is suggested that the value for this factor be put on a linear scale with a high- 
quality mission (such as Voyager or Apollo) being ten and a modest mission (such as 
Clementine) being one. This is not a scale from one to ten; missions could have values 
greater than ten or less than one. 

In order to use these scores in the algorithm, it will need to be normalized to unity. This can be 
accomplished by dividing all scores by the score of the highest ranked mission. 

The way cost should be entered into the equation was still up for debate. It was felt the best way was 
to have several (~3) bins of comparable cost, and once a mission was in a particular bin, 
cost would no longer be a discriminator against other missions in the same cost bin. 

Ed and PA are Educational Program Activities and Public Awareness 

Science Risk (SciRisk) 

This term should be expressed as something that approximates a probability (from zero to one). It is 
determined primarily from instrumentation and the quality of the science team. 

It was felt that Technology Infusion should not be given a separate score. The rationale for this is that 
the use of the new technology should already manifest itself by increasing the science or 
reducing the risk and would thus get into the score with an appropriate weight. 
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The final score should be a combination of ROI and Risk. 

Score = ROI x Risk 

Risk = Management Risk x CostRisk x Approach Risk 

Like science risk, these three terms should be expressed on a scale from zero to one and should be in 
the form of a probability. These terms will describe the probability that the area will cause 
the anticipated Return on Investment not to be achieved. 

The question of targeting the AO to a specific area was considered. The consensus was that if NASA 
determines in advance that there are reasons to do so, they should so state the areas of 
emphasis in the AO, but that the emphasis should not be exclusionary. The ability of the 
selection officer to exercise discretion when a clear case could be made was thought to be 
able to deal with most situations. 



Technology vs. Risk: Mission Success and Other Factors 
— D. Roalstad 
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The thrust of this splinter session was to identify those technology vs. risk issues as they relate to 

mission success and other factors. First we had to define what constitutes mission success 
and the other factor elements. These elements include technology transfer, technology 
infusion, SB/SDB infusion, education, and public awareness. 

A process established for the session was to list all technology vs. risk issues relating to mission 

success and other factors we could think of. Reviewing these issues with at least a subjec- 
tive assessment of priorities, we developed a preliminary set of recommendations. These 
issues and recommendations were then presented in a plenary session to the fall workshop 
An attempt has been made in this report to capture the comments received, but I’m sure 
more thoughts will be generated in the months ahead. 

Following are charts summarizing the session results: 


DEFINITIONS AND SCOPE 

Mission Success is determined by accomplishing the floor science defined by the proposing PI. 
Floor Science is defined as the minimal science the PI is willing to fly. 

OTHER FACTORS 


❖ Technology Transfer 

❖ Technology Infusion 

❖ SB/SDB Requirements 

❖ Education 

❖ Public Awareness 


ISSUES 

Take risk but don’t fail perception is a conflict. 

Technology demonstration before science mission — is that necessary? 
How much should science missions push the technology envelope? 
What reliability level is adequate for mission sources? 

How is science downgrade vs. risk treated? 

Go/No Go vs. 2% weighting factor on other factors. 

Other factors — goal vs. objective. 

Reasonable levels for other factors. 

Acceptable launch risk level. 
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ISSUES DISCUSSION 

❖ There is a perception that the agency cannot accept any failure that is in conflict with 
statements to accept higher risk levels. This environment results in a gap between 
proposers and reviewers on what is acceptable — very subjective at best. 

❖ There is a growing concern that technology must be demonstrated (e.g.. New Millennium) 
before use on a science mission. Science by its very nature tends to utilize new technology 
that is unique to the Pi’s science, which has been underdevelopment in university labora- 
tories. Technology output has been one of the best added-value products of science 
missions. 

❖ Reasonable reliability levels of parts subsystems and systems can be prescribed and 
achieved for science programs to mitigate risk. 

❖ Science objectives can be reduced from the desired level to the floor with a logical plan 
should risk assessments dictate. The PI now has responsibility and authority to manage 
with firm guidelines. 

<♦ The AO implies that there is a 2% (approximately) weighting factor applied to other 
factors, yet the perception is that a Go/No Go criterion is applied to the proposal evalua- 
tion. 

❖ Other factors are currently defined as objectives for the proposers with excessive levels. 
Would goals be more appropriate with reasonable levels? 

❖ Launch risk levels are as important to the mission as is the space segment hardware — what 
is acceptable? 


RECOMMENDATIONS 

❖ Mission success should be defined as accomplishing the floor science established by the 
PI. The PI will propose science levels for accomplishment that are in excess of the mini- 
mum below which he would not fly. Therefore, the “floor” level of science is the minimal 
level of science the PI believes to be scientifically justifiable for flight. Should he not be 
able to achieve this level of performance within other program guidelines, he would 
terminate the program. 

❖ Discovery Mission should be science driven, not technology driven. However, new tech- 
nology is generally essential to the science objectives proposed. New technologies pro- 
posed should be identified and a development plan provided that will bring these technolo- 
gies to level 6 by phase C/D start and level 8 by launch. (Ref. NASA Code R document 
for technology development state.) Launch services should demonstrate these same levels. 

❖ Other factor elements are generally a result of social-political mandates on the program 
and can be managed. The consensus, however, was that the SB/SDB levels of 8%/8% for 
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a total of 1 6% of program value is excessive and that all other factors should be estab- 
lished as goals rather than firm objectives. All functions, as currently mandated, can be 
accomplished at added expense to the program and could detract from the primary science 
focus. For maximum efficiency on achieving lowest cost we want high focus and the 
shortest possible schedule. 

Cost growth is the risk element in having to perform other factors where minimal experi- 
ence exists for levels that are significantly above those previously experienced. For one-of- 
a-kind types of missions, the technology problems are most difficult to manage, requiring 
highly focused team attention. Forcing work into the SB/SDB adds to subcontracts 
management and communication burden as a schedule and cost risk. 

The bottom line from our session was that other factors could be managed as is with some 
cost and schedule risk to the program. The preference was to recommend goals rather 
than objectives and to reduce SB/SDB requirements to more reasonable levels; 8-10% 
total. Perhaps a more efficient process can be established that doesn’t place an abnormal 
burden on the science team that is desperately trying to maintain focus. 
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Cost 

— G. S. Hubbard 


Note on Models 


Models useful for sensitivity and risk analysis for: 

❖ a given mission 

❖ by a given team 

❖ a specific management plan 

❖ similar system requirements 

When making comparisons between different proposals the modeler is forced to make many 
subjective estimates. 

Note on Cost Proposal Instructions 


The devil is in the details — small changes in wording can greatly change the amount of work 
required with little added value. 


Suggested weighting: 


Science 

Technical 

Management 

Cost Risk 

Opportunity 

Cost 


Step One 

70 

20 

10 

0 

N/S 

Categorized 


Step Two 

20 

25 

25 

30 

N/S 

Not Scored 


Step Two Evaluation 

❖ Evaluation criteria — science, technical, management, cost risk ( absolute cost is not 

scored) 

❖ All proposal evaluated together WRT same criteria 

❖ Evaluate cost risk by analysis of proposal data (do not evaluate on basis of probable cost 
models) 

❖ Questions and orals are a required part of the process to ensure complete understanding 
of the cost risk 

❖ Rank proposals and submit to source selection authority (SSA) 


SSA has ranked list with cost for each. 
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Suggested Two-Step Process 
Step One 

❖ Open solicitation — purpose to narrow field 

❖ Limits on mission cost, launch date, other program ground rules 

❖ Primarily evaluated on value of science WRT published NASA goals 

❖ Proposals grouped in 3 or 4 cost bins 

❖ Best 2 or 3 proposals in each bin selected for step 2 

❖ Cost proposal consists of total mission cost estimate at second level WBS spread by year 
and assumptions and rationale to support of basis of estimate 

Step Two 

❖ Evaluation weighted differently than step 1 

❖ Cost proposal consists of detailed phase A cost estimate and priced options for phases B, 
C/D, E 

❖ Priced options consist of SOW, WBS to level 3 (subsystem), summary BOEs at level 3, 
time phased by year, manpower spread, and no 1411 

Goals 

❖ Reduce LCC of the AO 

❖ Expand science opportunity by reducing cost of entry 

❖ Provide equal opportunity for missions of different size 

❖ Provide alternative to should cost models for evaluation 
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AO! AO Process Changes and Program Management 
— L. Mitchler 


AO/AO Process/Program Management 

»:♦ 1 4 attendees (broad representation) 

❖ Defined charter 

❖ Identified 21 issues 

❖ Synthesized to 7 problems 

♦ What problem will we solve? 

❖ Top problems discussed 

♦ Life cycle cost of AO 

♦ Science in the hands of scientists 

♦ AO clarity 

♦ Big and small missions 

»> “Solutions” discussion begun 

Problems/Recommendations 

1 . How to reduce “life cycle cost of AO”? 

All resources (t & $) and still get best proposals. 

2. How to clearly communicate the intent of AO? 

Requirements 

Evaluation/scoring/criteria (RFP vs. AO) 

Simplify 

Including support services (DSN) 

3. Ensure AO reflects complex strategy and connect to other programs. 

4. Ensure science/ROI expressed in next AO 

5. Selection IPDT — e.g., DSN assets common loop 

6. Should NASA revisit cap on “contributed costs”? 

Commercial 
International 
United States 

7. Upfront LA spending to minimize later problems 

Meet funding profile 
Guideline available funds 

Problem: Big Mission vs. Small Missions 

Should there be different AO categories? 


Assumption: There are different and inherent complexity and cost missions. 
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Proposed. Solutions: 

1 . NASA state maximum cost to be spent on given AO defined/constant boundary conditions 
well before AO. 

2. Separate cost category AOs. 

❖ Adapt to changing environment 

♦ Mercury 

♦ Lander 

♦ Rendez 

♦ Lunar 

♦ NEO 

❖ Big vs. small missions 

❖ During selection process 

❖ Cost/risk 

❖ Cost/science ROI 

Problem: Life cycle cost of AO l Science in hands of science 

Proposed Solutions: 

1 . Science validation. 

Compulsory 
I-U + concern cat 
Pub results and definitions 
Not a screening 

Pros: 

Science/Science 
Early Validation 
Doesn’t Screen 

Cons: 

Bidding War 
Postpones Teams 
Doesn’t Screen 

Process Solutions 

1. No substantive change/constant criteria two step 

2. Science validation 

3. Two-step downselect 

Pros: 

Reduces full proposal cost 
Get best science 

Cons: 

Too long? 

Encourages buy in? 

Decouples from cost? 

Still require some study investment 

❖ AO only 

❖ Based on firm science floor 

❖ NTE cost 
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Summary 

❖ More time required for definitive recommendation. 

❖ Whatever we do should maintain cognizance of cost-constrained environment. 

❖ Watch the Midex experiment. 


DISCOVERY LESSONS LEARNED 

Discovery Philosophy 

❖ Keep “Faster, Better, Cheaper” Theme 

❖ Keep average cost/mission < $80M 

❖ Keep mission plans simple 

❖ Keep payload <3-4 instruments 

❖ Let other Code S programs do more costly missions 

❖ Do not have 2 or 3 cost classes 

Announcement of Opportunity 

❖ Delete repetition 

❖ De -emphasize “Opportunity” — let us do our jobs without being “politically correct” 

❖ Do not expect so much for a Phase A study 

❖ Do not expect “ROM costs” to be “not to exceed” 

❖ Do not change rules, i.e., ROM + 1 5 % became ROM 


Evaluation 

❖ Was “fair and equitable” 

❖ Leave it as is 

❖ Do it on time 
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